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Abstract 


Control  of  total  ownership  cost  (TOC)  is  a  continuing  initiative  to  manage 
costs  over  the  entire  life  cycle  of  a  weapon  system.  There  are  several  major 
categories  of  costs  that  contribute  to  Total  Ownership  Cost  but  the  principal 
categories  are  (1)  R&D,  (2)  Production,  (3)  Operating  and  Support,  and  (4)  Disposal. 
System  TOC  is  the  same  as  Life  Cycle  Cost  (LCC)  and  must  be  planned  and 
controlled  from  requirements  definition,  system  development,  and  sustainment — 
focusing  on  affordability,  and  cost  to  achieve  required  operational  availability.  The 
Program  Manager  (PM)  is  responsible  for  managing  system  TOC,  with  input  from 
key  stakeholders,  such  as  the  sponsor  and  users.  This  paper  updates  our  work  in 
2003,  addressing  initiatives  to  control  cost,  congressional  pressure  to  control  cost, 
leadership  guidance,  controls,  and  incentives  that  can  be  employed  to  encourage 
emphasis  on  system  affordability.  There  is  some  discussion  of  metrics  to  control  life 
cycle  cost  and  the  requirements  for  databases  to  assist  in  estimating  and  comparing 
life  cycle  costs.  The  growing  cost  of  software  supportability  and  its  impact  on  system 
TOC  is  also  discussed,  along  with  methodologies  for  reducing  these  costs. 

Keywords:  affordability,  research  and  development  cost,  total  ownership  cost 
(TOC),  reduction  in  total  ownership  cost  (RTOC),  life  cycle  cost,  operating  &  support 
cost,  sustainment  cost,  developmental  cost,  production  cost,  software  supportability, 
post  deployment  software  support  (PDSS) 
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I. 


Background 


In  2003,  we  prepared  a  report  on  the  Reduction  of  Total  Ownership  Cost.  At 
the  time  there  was  substantial  effort  ongoing,  and  in  more  than  seven  years  since, 
much  has  happened.  As  is  always  the  case  with  a  very  large  organization  such  as 
the  Department  of  Defense  (DoD),  change  has  come  slowly  and  unevenly  across 
the  organization.  In  201 1 ,  we  see  clearly  a  number  of  things  that  need  to  be  done  to 
control  and  reduce  the  life  cycle  cost  of  weapon  systems.  Many  of  the  steps  that 
need  to  be  accomplished  are  well  known  but  not  practiced.  Some  aspects  are  not 
widely  understood  or  are  otherwise  not  in  place.  We  begin  this  report  with 
background  extracted  directly  from  our  2003  report.  In  some  cases,  we  have  made 
minor  edits  to  update  terminology. 

The  focus  of  our  current  research  can  be  found  in  Chapters  2  through  6.  Our 
conclusions  and  recommendations  are  located  in  Chapter  7. 

A.  Definitions  (Boudreau  &  Naegle,  2003,  p.  1) 

Total  Ownership  Cost  (TOC)  has  two  definitions;  the  first  is  very  broad, 
looking  from  the  DoD  or  Service  perspective. 

DoD  TOC  is  the  sum  of  all  financial  resources  necessary  to  organize,  equip, 
train,  sustain,  and  operate  military  forces  sufficient  to  meet  national  goals  in 
compliance  with  all  laws,  all  policies  applicable  to  DoD,  all  standards  in  effect 
for  readiness,  safety,  and  quality  of  life,  and  all  other  official  measures  of 
performance  for  DoD  and  its  Components.  DoD  TOC  is  comprised  of  costs  to 
research,  develop,  acquire,  own,  operate,  and  dispose  of  weapon  and 
support  systems,  other  equipment  and  real  property,  the  costs  to  recruit,  train, 
retain,  separate  and  otherwise  support  military  and  civilian  personnel,  and  all 
other  costs  of  business  operations  of  the  DoD.  (Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  [USD(AT&L)],  1998,  p.  ) 

Much  of  the  activity  described  in  this  definition  is  beyond  the  capability  of  a 
weapon  system  program  manager  to  influence.  However,  it  is  deliberately  broad  in 
scope  to  include  the  many  different  possibilities  for  various  stakeholders  to  reduce 
ownership  cost. 
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The  second  definition  is  deliberately  written  from  the  vantage  point  of  the 
program  manager  (PM)  of  the  warfighting  system. 

Defense  Systems  TOC  is  defined  as  Life  Cycle  Cost  (LCC).  LCC  (per  DoD 
5000.4M)  includes  not  only  acquisition  program  direct  costs,  but  also  the 
indirect  costs  attributable  to  the  acquisition  program  (i.e.,  costs  that  would  not 
occur  if  the  program  did  not  exist).  For  example,  indirect  costs  would  include 
the  infrastructure  that  plans,  manages,  and  executes  a  program  over  its  full 
life  and  common  support  items  and  systems.  The  responsibility  of  program 
managers  in  support  of  reducing  DoD  TOC  is  the  continuous  reduction  of 
LCC  for  their  systems.  (USD[AT&L],  1998,  p.  2) 

As  Dr.  Gansler  said  in  his  1998  memorandum  from  which  the  above 
definitions  were  extracted,  the  PM’s  job  in  trying  to  reduce  TOC  is  a  very  difficult 
one,  and  PMs  should  seek  help  wherever  they  can  to  reduce  ownership  costs. 
Because  of  the  extreme  amount  of  focus  on  the  authorized  and  appropriated  budget, 
it  is  easy  for  PMs  to  likewise  focus  on  the  near-term  acquisition  cost  and  make 
decisions  that  appear  to  be  beneficial  in  reducing  acquisition  costs  but  that  are 
detrimental  to  operations  and  support  costs  because  they  increase  future  budgets. 
For  example,  if  a  program  experiences  a  budget  cut,  which  is  a  very  typical 
occurrence,  there  is  significant  pressure  to  continue  to  deliver  the  same  number  of 
new  systems,  even  though  there  has  been  a  cut  in  funding.  As  a  result,  the  PM 
looks  for  something  else  to  cut  out  of  the  program.  Logistics  performance  items  are 
rarely  deemed  to  be  Key  Performance  Parameters  (KPPs),  so  they  become  easy 
targets  for  cutting  during  budget  cut  drills.  So  the  PM  is  faced  with  a  choice:  cut  the 
number  of  systems  to  be  acquired,  or  reduce  the  logistics  performance  (eliminate 
Built-in  Test  [BIT]  capability,  onboard  diagnostics/prognostics/autonomics,  etc.), 
which  will  add  significant  O&S  costs  well  after  the  PM  has  moved  to  a  different 
position.  Which  choice  do  you  suppose  is  most  appealing  to  the  PM? 

Even  the  definition  of  system  is  changing  as  we  move  to  system-of-systems 
(SoS)  and  net-centric  system  concepts.  For  example,  developing  the  Single 
Integrated  Air  Picture  (SIAP)  as  a  system  means  that  all  the  Services’  manned  and 
unmanned  aircraft,  many  guided  and  unguided  missile  platforms,  and  a  host  of 
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command  and  control  systems  are  now  at  least  a  part  of  SIAP.  Changes  to  any  of 
the  platforms  (especially  software  changes)  could  result  in  changes  to  others, 
impacting  the  TOC  of  the  individual  weapon  platforms  and  of  the  SIAP  system.  How 
do  we  account  for  SoS  or  net-centric  system-driven  changes/maintenance  in 
forecasting  or  even  attributing  cost  elements?  For  example,  consider  the  networking 
software  for  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  as  updated  for 
the  Ml  Main  Battle  Tank  and  several  other  platforms  using  the  network.  The 
software  update  did  not  integrate  perfectly  with  the  Mi’s  onboard  software  suite, 
causing  uncommanded  turret  movement  (Federation  of  American  Scientists  [FAS], 
2011a).  Because  the  other  platforms  in  the  system  did  not  experience 
interoperability  problems,  the  PM  for  Ml  was  assigned  responsibility  for  the 
diagnosis  and  repair  of  Ml  software  to  be  compatible  with  the  FBCB2  update.  This 
begs  the  questions  of  which  program  to  attribute  the  TOC  expense  (FBCB2  or  Ml ), 
how  would  such  TOC  factors  be  forecasted,  and  by  whom  would  they  be  budgeted? 

B.  TOC  Processes:  CAIV  and  R-TOC  (Boudreau  &  Naegle,  2003, 

P-2) 

Pursuit  of  Total  Ownership  Cost  reduction  at  the  level  of  the  warfighting 
system  may  be  separated  into  two  major  approaches  that  are  connected,  end-to- 
end,  along  a  life-cycle  time  line.  During  the  developmental  phases,  the  effort  or 
process  is  called  Cost  As  an  Independent  Variable  (CAIV).  For  systems  in  the  field 
or  fleet,  the  process  or  goal  becomes  Reduction  of  Total  Ownership  Cost  (R-TOC). 
The  chart  in  Figure  1  is  a  typical  depiction  of  the  CAIV/R-TOC  relationship. 
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Figure  1.  CAIV/R-TOC  Relationship 

(Kaye,  Sobota,  Graham,  &  Gotwald,  2000,  p.  354) 

The  first  approach,  CAIV  addresses  TOC  during  the  warfighting  system’s 
developmental  phases,  beginning  with  the  Concept  Refinement  phase.  The  focus  of 
CAIV  is  to  establish  cost  targets  based  on  affordability  and  requirements  and  then  to 
manage  to  those  targets,  thereby  controlling  TOC.  CAIV  includes  consideration  of 
costs  for  development,  production,  operations  and  support,  and  disposal.  An 
example  of  the  CAIV  process  would  be  to  set  specific  cost  and  reliability  targets  for 
each  subsystem  or  component  of  a  weapon  system  in  development  such  that  the 
warfighting  system  would  be  able  to  achieve  the  required  operational  availability  (Ao) 
at  the  specified  cost. 

Employing  the  CAIV  concept  early  in  the  developmental  process  offers, 
potentially,  the  greatest  opportunity  for  TOC  reduction  at  the  lowest  possible 
investment  cost.  As  an  example,  the  TOC  impact  of  using  two  different  power  plants 
presents  an  opportunity  to  use  the  CAIV  evaluation  technique  to  estimate  the  TOC 
impact  and  make  a  best-value  decision.  For  illustrative  purposes,  consider  a 
standard  internal  combustion  engine  at  a  cost  of  $7,500  versus  a  hybrid-electric 
power  plant  costing  $19,000.  The  impact  to  the  acquisition  cost  is  evident,  but  it 
excludes  the  cost  savings  associated  with  fuel  consumption  over  the  life  of  the 
system.  If  the  system’s  operational  mode  indicates  an  average  usage  of  15,000 
miles  per  year  and  an  economic  useful  life  (EUL)  of  20  years,  the  total  miles 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  6  - 

NAVAL  POSTGRADUATE  SCHOOL 


expected  would  be  300,000.  If  the  standard  engine  in  our  comparison  is  estimated 
at  10  miles  per  gallon  and  the  hybrid  engine  is  estimated  at  25  miles  per  gallon,  the 
estimated  fuel  saved  by  the  hybrid-powered  system  would  be  18,000  gallons.  At  a 
current  estimate  of  $1 .25  per  gallon,  the  operating  and  support  impact  is  $22,500 
per  system  (TOC  improvement:  $1 1 ,000  less  expensive  than  the  standard  engine), 
and  there  are  other  reductions  in  fuel  supply  assets  and  attendant  personnel  that 
apply. 


The  second  approach  to  TOC  is  the  R-TOC  and  focuses  on  the  reduction  of 
average  procurement  unit  cost  (APUC)  and  weapon  system  sustainment  cost  (i.e. , 
operating  and  support  [O&S]  costs).  R-TOC  is  employed  as  the  warfighting  system 
is  produced  and  placed  in  service.  Examples  of  R-TOC  would  be  a  value 
engineering  change  proposal  (VECP)  to  reduce  the  cost  of  manufacturing  a 
component  by  improving  the  process  yield  (the  percentage  of  the  manufactured 
items  that  are  defect  free)  or  a  VECP  to  reduce  the  operating  and  support  cost  by 
improving  the  reliability  of  an  expensive  subsystem  or  component.  Often  there  are 
the  secondary  benefits  of  enhanced  performance  (i.e.,  improved  reliability  and 
operational  availability),  but  the  forcing  function  is  the  reduction  of  operating  and 
support  costs,  the  largest  constituent  of  TOC. 

System  software  has  become  an  ever-increasing  TOC  driver  the  more 
systems  rely  on  software  functions. 


C.  TOC  Obstacles  (Boudreau  &  Naegle,  2003,  pp.  4-7) 

Someone  who  is  not  involved  with  program  management  might  wonder  what 
is  especially  difficult  about  containing  and  controlling  TOC.  In  truth,  there  are  many 
difficulties.  What  follows  is  a  description  of  some  of  the  obstacles  that  get  in  the  way 
of  controlling  or  reducing  TOC.  All  of  these  obstacles  are  well  known  but  are 
entrenched  and  difficult  to  overcome. 


The  competing  interests  of  users,  developers,  prime  contractors, 
subcontractors,  the  Office  of  the  Secretary  of  Defense  (OSD),  Service  headquarters, 
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maintainers,  buying  commands,  and  Congress  may  negatively  impact  TOC.  The 
“user”  who  establishes  requirements  for  a  new  system  may  be  transfixed  by  the 
technical  performance  and  may  not  clearly  establish  requirements  for  ownership 
cost  to  achieve  specified  system  availability.  Materiel  developers  may  be  too 
focused  on  acquisition  cost  and  schedule  (a  typical  complaint  from  the  user 
community)  and  may  ignore  future  logistics  support  issues.  Prime  contractors  may 
concentrate  on  production  costs,  with  less  regard  for  system  sustainment  costs, 
particularly  if  their  contract  directs  them  toward  reduction  in  production  costs  or  if 
they  sense  that  their  customer  is  not  interested  in  sustainment  issues.  The  OSD 
and  Service  headquarters  may  encourage  poor  TOC  decisions  through  funding 
instability  and  failure  to  demand  life-cycle  affordable  solutions.  Maintainers  may 
contribute  to  poor  R-TOC  by  failing  to  speak  out  loudly  on  lessons  learned  from 
previous  systems.  Buying  commands  may  contribute  to  increased  ownership  costs 
by  failing  to  look  aggressively  for  cost  drivers  that  need  to  be  redesigned  for  lower 
cost  of  operation  and  improved  reliability.  The  Congress  may  restrict  R-TOC  by 
constraining  the  choices  of  cost-effective  sustainment  approaches. 

Balancing  Total  Ownership  Cost  Goals  That  Are  Conflicting.  Successful 
program  management  includes  the  ability  to  achieve  balance  within  a  program. 
Indeed,  PMs  are  directed  by  DoDD  5000.1  to  manage  their  programs  in  a  balanced 
way  (USD[AT&L],  2003,  Enel.  1,  para.  El. 29).  Facets  and  perspectives  that  need  to 
be  balanced  are  manifold.  Four  elements  of  TOC  that  require  balancing  are 
development  costs,  procurement  costs,  operating  and  support  costs,  and  disposal 
costs.  Development  costs,  the  expenditure  of  resources  during  system 
development,  may  pay  off  in  terms  of  reduced  production  and/or  sustainment  costs; 
producibility  studies  may  save  significant  manufacturing  costs;  and  reliability  testing 
early  in  a  program  may  allow  for  avoidance  of  sustainment  costs  over  the  service  life 
of  the  weapon  system.  Occasionally,  procurement  or  production  cost  constraints 
may  conflict  with  sustainment  cost  targets;  for  example,  heavy  pressure  to  reduce 
production  costs  may  lead  to  the  selection  of  components  that  are  inexpensive  but 
not  reliable.  Such  choices  would  reduce  production  cost  but  increase  sustainment 
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costs  and  very  possibly  result  in  an  increase  of  TOC.  When  such  cost  goals  conflict, 
a  reasonable  metric  for  maintaining  balance  would  appear  to  be  minimization  of 
TOC  (i.e.,  life  cycle  cost,  but,  often,  TOC  is  sub-optimized  due  to  these  competing 
pressures). 

Balancing  Cost,  Schedule,  System  Performance,  Sustainment,  Quality,  and 
Risk.  In  the  same  way  that  ownership  cost  goals  must  be  balanced  and  harmonized, 
system  solutions  must  be  found  that  balance  TOC  against  procurement  cost  goals, 
program  schedule  goals,  system  technical  performance,  equipment  quality, 
supportability  performance,  and  availability. 

The  DoD  is  relying  on  sophisticated,  software-intensive  systems  to  improve 
survivability  and  lethality,  but  software  is  susceptible  to  high  TOC.  Software 
“maintenance”  is  becoming  a  major  TOC  driver  (Naegle,  2004,  p.  1).  Software  is 
difficult  to  accurately  estimate  and  sensitive  to  changing  requirements.  Its 
complexity,  interface  requirements,  and  relative  ease  in  adding  capability  also  tend 
to  make  it  maintenance  intensive  (Humphrey,  1990,  Chapter  4).  Software’s  negative 
influence  on  TOC  is  exacerbated  by  the  fact  that  software  support  is  most  often 
provided  by  contractors,  with  very  little  opportunity  to  move  software  support  to 
Government  sources.  For  example,  the  1980s  vintage  B1 B  Bomber  budgets  were 
approximately  $100  million  annually  for  software  maintenance,  and  the  2010  budget 
was  $227  million  because  several  software-intensive  systems  were  also  being 
upgraded  (Naegle  &  Petross  a.,  2010,  p.  25).  The  Bl’s  software  support  is  achieved 
through  both  contractor  and  Government  software  support  organizations  and  is 
coordinated  by  an  Air  Force-supported  Program  Management  Office  (PMO). 


During  each  life-cycle  phase,  the  approach  to  TOC  reduction  and  the 
methodology  may  change  somewhat,  while  ownership  cost  goals  and  targets 
become  more  refined.  For  example,  tradeoff  processes  used  in  the  Materiel 
Solution  Analysis  phase  may  be  beneficial  during  that  phase  but  may  be  inadequate 
for  the  Engineering  and  Manufacturing  Development  (EMD)  phase  without  the 
inclusion  of  specific  contractor  incentives. 
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Materiel  Developer  Instability.  Key  members  of  the  materiel  developer  team 
change  over  time.  For  example,  the  PM  during  the  Integrated  System  Design  phase 
would  be  unlikely  to  remain  in  that  position  through  the  Production  and  Deployment 
phase.  As  key  personnel — PMs,  chief  engineers,  product  support  managers, 
business-financial  managers — change,  program  emphasis  shifts,  at  least  subtly. 
These  personnel  changes,  which  are  a  fact  of  life,  may  reflect  in  program  missteps, 
including  missed  TOC  targets. 

Funding  Instability.  Resources  tend  to  be  unstable  and  subject  to 
unanticipated,  unexpected  changes.  Funding  instability  is  also  a  fact  of  life  in 
Government  acquisition  programs.  Each  time  that  funding  is  cut  from  a  program, 
decision-makers  adjust  the  program  by  postponing  or  eliminating  some  activity  or 
system  attribute.  Decisions  are  made  that  will  keep  the  program  viable,  and  often 
the  choice  is  to  omit  a  system  feature  or  a  near-term  activity  that  will  reflect 
negatively  on  TOC — but  not  until  later.  Easing  back  on  O&S  cost  targets  is  a 
tempting  sacrifice  when  program  funding  gets  cut.  For  example,  reliability-centered 
maintenance  studies  cut  to  reduce  cost  during  EMD  would  not  affect  the  program 
noticeably  until  later  on,  when  operational  systems  are  in  the  field  or  fleet;  the 
associated  effect  on  TOC  might  be  substantial.  Eliminating  onboard 
diagnostics/prognostics  would  certainly  help  meet  funding  cuts  during  the 
Procurement  phase,  but  would  likely  be  extremely  costly  in  terms  of  maintainer 
training,  diagnostics  time,  erroneous  fault  isolation,  errant  parts  ordering,  and 
associated  maintenance  man-hours  for  the  life  of  the  system. 

Sticker  Shock.  The  fact  that  a  system’s  TOC  “price  tag”  is  extremely  high 
when  compared  to  its  contract  unit  price  may  tend  to  keep  stakeholders  from 
discussing  TOC  in  any  open  forum,  fearing  that  “sticker  shock”  might  cause  an 
adverse  reaction  from  a  decision-maker  or  politically  powerful  individual  accustomed 
to  seeing  much  lower  cost  figures.  As  an  example,  consider  a  system  with  an 
average  procurement  unit  cost  (APUC)  of  $1 .5  million  and  a  program  acquisition 
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cost  of  $2  million.1  Typically,  program  acquisition  cost  would  represent  only  about 
28%  of  each  individual  system’s  TOC,  with  the  remaining  72%  representing  O&S 
and  disposal  costs  of  about  $5  million,  for  a  TOC  of  $7  million  per  each  weapon 
system.  With  an  acquisition  objective  of  2,000  systems,  the  total  procurement  cost 
would  be  $3  billion,  with  a  TOC  estimate  of  approximately  $14  billion.  If  unfamiliar 
with  TOC  estimates  and  without  a  readily  available  basis  for  comparison,  a  decision¬ 
maker  might  mistakenly  conclude  that  the  system  would  be  unaffordable  and  cancel 
the  program.  Concern  for  such  a  scenario  may  create  an  impediment  to  widespread 
use  of  system  life  cycle  cost  numbers,  which  would  have  the  effect  of  refocusing 
decisions  onto  the  acquisition  “price  tag,”  not  the  TOC  “price  tag.” 

D.  Management  of  TOC 

There  is  an  increasing  body  of  knowledge  related  to  the  control  of  TOC.  In 
addition  to  specific  congressional  direction  and  ever  more  detailed  DoD  direction, 
very  thoughtful  articles  have  been  published  on  the  matter.  Every  PM  tries  different 
approaches  to  reduce  costs.  Additionally,  commercial  best  practices  have  been 
recognized  and  suggested  for  use  within  the  DoD  (e.g.,  GAO-03-57  [GAO,  2003]  in 
its  entirety). 


1  APUC  is  the  total  procurement  cost  divided  by  the  total  procurement  quantity.  Program  acquisition 
cost  includes  APUC,  facilities,  RDT&E,  and  other  procurement  costs.  These  terms  are  discussed  in 
more  detail,  beginning  on  page  24. 
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The  Focus  of  this  Paper — Total  Ownership 
Cost  in  2011 


A.  Purpose 

The  purpose  of  this  working  paper  is  to  gather  together  the  various 
approaches  for  controlling  and  reducing  TOC  and  to  describe  tools  and  methods  to 
assist  PMs  and  others  in  addressing  TOC  more  effectively. 

B.  Scope  of  This  Study 

This  study  examines  TOC  from  the  perspective  of  congressional  direction,  the 
perspective  of  the  OSD  and  Service  leadership’s  governance,  the  perspective  of  PM 
execution,  and  the  perspective  of  available  infrastructure  support. 

C.  Introduction 

This  report  extends  our  research  that  was  first  published  in  2003.  At  that 
time,  just  as  currently,  there  was  significant  attention  being  paid  to  TOC.  There  were 
a  number  of  initiatives  collected  and  shared  on  a  TOC  website  constructed  by  the 
Institute  for  Defense  Analyses  (IDA;  www.ida.org).  Additionally,  the  DAU  Acquisition 
Community  Connection  website 

(https://acc.dau. mil/CommunitvBrowser.aspx?id=22509&lanq=en-US)  also  contains 
useful  approaches  to  TOC  and  R-TOC.  Looking  over  the  TOC  landscape  in  2003, 
one  would  not  conclude  that  there  was  a  shortage  of  ideas  related  to  reducing  TOC. 
The  same  appears  true  today — there  are  lots  of  useful  approaches  for  reducing 
TOC,  or  weapon  system  life  cycle  costs,  reflecting  the  increasing  anxiety  over 
skyrocketing  costs  of  ownership.  Many  aspects  of  Defense  acquisition  have 
continued  to  evolve,  making  it  difficult  to  know  what  has  helped  to  control  costs  and 
what  may  have  had  the  opposite  effect  or  had  no  significant  effect.  The  following 
paragraphs  provide  a  few  examples  to  help  make  the  point. 
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There  are  increased  acquisition  reviews  (USD[AT&L],  2008c).  PMs  and 
those  working  in  program  offices  know  that  reviews  are  expensive  and  divert 
attention  from  other  management  activities.  Have  increased  reviews  contributed  to 
increased  cost  or  reduced  it?  Has  developmental  cost  increased  while  the  larger 
sustainment  costs  decreased?  Does  anyone  really  know? 

Acquisition  reforms,  launched  in  the  mid  1990s,  resulted  in  many  changes  to 
the  way  we  do  acquisition  business.  For  example,  acquisition  programs  have 
reduced  their  preparation  for  sustainment.  MIL-STD-1388-2A  and  -2B  ,  which 
became  obsolete  under  the  Acquisition  Reform  initiatives  of  the  1990s,  were  very 
detailed  and  for  many  years  had  guided  acquisition  logistics  planning;  they  were 
mandatory  until  circa  1995. 2  These  standards  governed  supportability  analyses  and 
served  to  inform  sustainment  planning,  but  they  were  onerous  requirements  and 
sometimes  resulted  in  analyses  that  languished  on  the  shelf  and  were  never  put  to 
use.  Did  the  discontinued  use  of  these  standards  result  in  the  de-emphasis  and  de¬ 
funding  of  rigorous  sustainment  planning,  in  turn  causing  an  increase  in  the  cost  of 
sustainment  and  a  corollary  reduction  in  warfighting  system  readiness? 

Another  acquisition  reform  initiative  during  the  mid-1990s  created  a  bias 
against  purchasing  technical  data  packages  (TDPs).3  Did  that  result  in  the 
avoidance  of  unnecessary  and  unneeded  TDPs,  or  might  this  initiative  have 
prevented  the  purchase  of  technical  data,  leaving  a  program  with  few  good  options 
related  to  re-buys  and  purchase  of  repair  parts?  Did  it  narrow  the  range  of  choices 
related  to  component-  and  system-level  maintenance? 


2  In  the  mid-1990s,  there  were  numerous  Acquisition  Reform  initiatives  intended  to  streamline 
acquisition  processes  and  reduce  cost.  One  of  these  initiatives  was  “specs  and  standards”  reform. 
Many  Government  specs  were  rescinded  to  reduce  the  Government  burden  and  cost  of  maintaining 
specs;  in  many  cases,  the  Government  switched  to  commercial  specifications  that  were  maintained 
by  various  technical  societies  or  associations.  Other  mandatory  specs  were  rescinded  because  they 
were  thought  unnecessary  or  provided  insufficient  benefit  for  the  cost  expended.  MIL-STD  1388-2A 
and  -2B  were  thought  by  some  to  fall  into  the  latter  category. 

3  Another  Acquisition  Reform  initiative  was  avoiding  the  purchase  of  technical  data  packages  in 
support  of  new  systems. 
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Has  performance-based  logistics  (PBL) — mandated  in  the  DoD  by  the  QDR  in 
September  2001  and  implemented  in  2002  (USD[AT&L],  2002) — reduced  the  cost  of 
sustainment  or  has  it  increased  those  costs?  Coupled  with  early  tech  data  choices, 
have  logisticians  been  forced  into  choices  that  make  sustainment  more  expensive 
throughout  the  weapon  system’s  life  cycle  (Kratz  &  Buckingham,  2010)? 

First  Gut  Question.  Have  Acquisition  Reform  and  Acquisition  Excellence 
initiatives  removed  acquisition  controls  and  opened  up  an  array  of  poor  choices  for 
PMs  that  have  increased  system  life  cycle  costs  (LCC)?  Might  well-meaning 
Acquisition  Reform  and  Acquisition  Excellence  initiatives  have  offered  shortcuts  that 
have  ended  badly  (Kratz  &  Buckingham,  2010)? 

Second  Gut  Question.  Has  one  of  the  principal  problems  been  lack  of 
discipline?  In  our  2003  paper,  we  addressed  leadership  resolve  and  the  need  to 
speak  with  one  voice  about  affordability.  In  2003,  the  new  JCIDs  directives  did  not 
emphasize  affordability.  Today  those  directives  do  (For  example:  CJCS,  2009a, 
Enclosure  A,  paragraph  2-b  and  Enclosure  B,  paragraph  3-d;  CJCS;  2009b, 
Enclosure  G,  paragraph  1-d  and  Appendix  A  to  Enclosure  G,  paragraph  16; 

WSARA,  2009,  §  201 ).  Yet  one  must  ask,  do  user  study  groups  understand  their 
emerging  system’s  slice  of  mission  area  funding  over  its  life  cycle?  Do  users  take 
ownership  control  of  these  costs  by  establishing  key  performance  parameters 
(KPPs)  or  key  system  attributes  (KSAs)  for  O&S  cost  or  system  life  cycle  cost?  Do 
SoS  and  net-centric  system  PMs  understand  and  account  for  TOC  drivers 
associated  with  system  changes  (especially  software)  that  impact  system  platforms 
and  platform  changes  that  impact  overarching  systems?  Do  materiel  developers 
insist  on  clear,  unambiguous  sustainment  cost  goals  and  establish  solid,  well- 
reasoned  CAIV  targets?  Do  contractors  structure  their  developments  to  deliver 
warfighting  systems  that  meet  customer  cost  constraints?  A  dominant  problem 
might  be  discipline — cost  discipline — starting  with  the  OSD  and  Service  leadership 
and  including  users,  materiel  developers,  and  contractors. 
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Third  Gut  Question.  Is  ownership  cost  data  being  collected  and  placed  in 
databases  that  facilitate  analysis  and  comparison  to  ownership  cost  targets  such 
that,  program  by  program,  interested  parties  can  see  whether  DoD  programs  are 
performing  within  their  affordability  constraints?  Acquisition  leaders  must  be  able  to 
measure  cost  performance.  If  they  really  want  to  get  TOC  under  control,  O&S  cost 
must  be  sufficiently  accurate  and  detailed  that  it  can  be  used  to  suggest  where 
system,  subsystem,  or  component  improvements  are  needed. 
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Congressional  Intervention 


Interestingly,  the  questions  posed  in  Chapter  2  appear  to  have  been 
congressional  questions,  too.  Congress  already  seems  to  have  responded  to  an 
array  of  similar  concerns,  in  its  own  unique  way.  This  is  what  the  Weapon  System 
Acquisition  Reform  Act  of  2009  is  all  about.  This  is  what  Congress  is  addressing  in 
its  changes  to  Nunn-McCurdy.  This  is  what  motivated  Congress  to  require 
certificates  at  Milestones  A  and  B  (10  U.S.C.  §  2366a,  b).  This  appears  to  be  the 
congressional  motive  in  Public  Law  1 1 1-84  (National  Defense  Authorization  Act, 
2009),  which  institutes  product  support  managers.  Having  witnessed  a  lack  of  cost 
and  process  discipline  spanning  many  years,  particularly  in  the  area  of  sustainment 
costs,  Congress  has  acted  to  enforce  discipline,  instituting  procedures  with  force  of 
law  to  get  weapon  system  costs  under  control. 

A.  The  Weapon  System  Acquisition  Reform  Act  of  2009 

The  Weapon  System  Acquisition  Reform  Act  of  2009  is  a  congressional 
initiative  to  increase  rigor  in  development  of  DoD  Major  Defense  Acquisition 
Programs  (MDAPs).  The  principal  intent  seems  directed  at  controlling  the  ownership 
cost  of  the  DoD’s  warfighting  systems.  The  WSARA  advances  on  a  number  of 
different  fronts,  a  portion  of  which  are  described  in  the  following  paragraphs 
(WSARA,  2009). 

Director  of  Cost  Assessment  and  Program  Evaluation  (“Director  CAPE”).  The 
Director  CAPE  is  a  new  appointive  position,  devised  to  give  independent  advice  and 
analysis  to  SECDEF  and  DEPSECDEF  on  matters  that  fall  within  his  responsibility. 
Director  CAPE  is  responsible  for  functions  formerly  accomplished  by  Program 
Analysis  &  Evaluation,  the  Defense  PA&E  (WSARA,  2009,  §  101). 

Director  CAPE  has  two  deputies  (WSARA,  2009,  §  101): 

the  Director  for  Cost  Assessment  (formed  initially  from  the  Cost 
Analysis  Improvement  Group  [CAIG]),  and 
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■  the  Director  for  Program  Evaluation  (formed  from  the  remnants  of 
PA&E,  that  is,  PA&E  less  the  CAIG). 

Responsibilities.  Director  CAPE  is  responsible  for  cost  estimation  and  cost 
analysis  for  MDAP  acquisition  programs,  analysis,  and  advice  in  the  planning  and 
programming  phases  of  PPBE  (WSARA,  2009,  §  101).  Additionally,  Director  CAPE 
provides  analysis  and  advice  to  the  Joint  Requirements  Oversight  Council  (JROC) 
and  formulates  study  guidance  used  to  conduct  Analysis  of  Alternatives  of  new 
major  defense  acquisition  programs  (WSARA,  2009,  §  201 ).  These  responsibilities 
place  Director  CAPE  in  a  position  to  provide  advice  and  direction  related  to  the 
accuracy  of  acquisition  cost  estimates  and  the  affordability  of  acquisition  programs. 
Quite  apparently,  Director  CAPE  is  charged  with  advising  the  (JROC)  to  strengthen 
that  body’s  role  in  issues  of  cost  and  affordability,  as  noted  in  the  short  JCIDS 
discussion  (i.e.,  Second  Gut  Question)  in  paragraph  c  of  Chapter  II,  above. 

Cost  Estimation.  Director  CAPE  is  specifically  charged  by  Congress  with 
ensuring  the  accuracy  of  cost  estimation  and  cost  analysis  by  prescribing  policies 
and  procedures  specifically  related  to  acquisition  programs  (WSARA,  2009,  §  101). 

In  this  capacity,  Director  CAPE  is  required  to  provide  guidance  to  and  consult  with 
OSD  leadership  and  the  Secretaries  of  the  military  departments  regarding  specific 
cost  estimates  and  cost  analyses  to  be  conducted  for  a  major  MDAP  or  major 
automated  information  system  (MAIS)  program.  This  mandate  includes  specifics, 
such  as  selection  of  statistical  confidence  levels  of  cost  estimates  in  consideration  of 
life  cycle  costs  of  MDAP  and  MAIS  programs.  Director  CAPE  specifically  reviews 
independent  cost  estimates  (ICE)  for  the  Defense  Acquisition  Board  (DAB)  prior  to 
certifications,  LRIP,  or  FRP.  Director  CAPE  is  further  charged  to  review  cost 
analyses  and  records  for  MDAP  and  DAIS  and  is  given  authority  to  participate  in 
discussion  of  discrepancies  between  ICE  and  military  department  cost  estimates. 
This  includes  disclosure  of  statistical  confidence  levels  used  by  Director  CAPE  and 
the  Services.  Confidence  levels  below  80%  must  be  justified  and  included  in  the 
next  Selected  Acquisition  Report,  SAR,  which  is  sent  from  PMs,  through  their 
component  and  the  OSD,  to  Congress.  Director  CAPE  is  required  to  report  annually 
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on  cost-estimating  accuracy  and  compliance  with  policy,  along  with  consistent 
differences  in  cost-estimating  methodology  by  the  Services.  This  report  goes  to 
OSD  leadership  and  congressional  defense  committees  and  is  to  be  posted  on  the 
Internet  for  public  review. 

The  WSARA  specified  that  Director  CAPE  must  report  to  the  SECDEF  on 
O&S  costs  for  MDAPs  within  one  year  and  to  congressional  defense  committees 
within  30  days  thereafter,  followed  by  annual  reports  (WSARA,  2009,  §  101).  This 
represents  another  action  that  brings  focus  to  the  cost  of  operating  and  sustaining 
warfighting  systems.  This  requirement  has  caused  considerable  consternation.  See 
the  discussion  in  the  GAO  Report  10-717  section  of  Chapter  4  for  additional 
perspective  on  the  accuracy  of  O&S  cost  databases. 

Director  of  Program  Assessment  and  Root  Cause  Analysis  (Director  PARCA). 

The  WSARA  (2009,  §  103)  mandated  that  the  SECDEF  designate  a  senior  official 
responsible  for  conducting  program  assessment  and  root  cause  analysis  for  MDAPs. 
Director  PARCA,  is  responsible  for  evaluating  the  utility  of  performance  metrics  used 
to  measure  cost,  schedule,  and  performance  of  MDAPs  and  for  making 
recommendations  for  improvement.  This  individual  also  advises  on  MDAP 
performance  issues  prior  to  certifications  at  Milestones  A  and  B,  prior  to  Full  Rate 
Production  (FRP),  and  in  consideration  of  multi-year  procurement  decisions. 

Director  PARCA  accomplishes  root  cause  analysis  for  MDAPs  to  determine  causes 
for  shortcomings  in  cost,  schedule,  or  performance,  including  unrealistic 
performance  expectations;  unrealistic  baseline  estimates  for  cost  or  schedule; 
immature  technologies;  unanticipated  design,  engineering,  manufacturing,  or 
technology  integration  issues  in  program  performance;  changes  in  procurement 
quantities;  inadequate  funding  or  funding  instability;  or  poor  PM  performance 
(Government  and/or  contractor).  Director  PARCA  must  report  annually  (initially 
March  2010)  on  root  cause  analyses  for  MDAPs  and  submit  to  congressional 
defense  committees  a  report  of  activities  undertaken  during  the  preceding  year. 
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Director  of  Defense  Research  &  Engineering  (DDRE;  WSARA,  2009,  §  104). 
Together  with  the  Director  of  Developmental  Test  and  Evaluation  (Director  DT&E), 
DDRE  reviews  and  assesses  the  technological  maturity  and  integration  risks  of 
MDAPs.  The  WSARA  requires  annual  reports  (initially  March  2010)  to  the  SECDEF 
and  the  congressional  defense  committees.  This  strongly  encourages  MDAs,  PEOs, 
and  PMs  not  to  permit  programs  to  move  forward  until  they  are  technologically 
ready. 


Director  of  Systems  Engineering  (WSARA,  2009,  §  102).  This  director  is 
required  to  develop  policy  and  guidance  for  systems  engineering  master  plans  for 
MDAPs  in  support  of  life-cycle  management,  sustainability,  and  reliability  growth  in 
contractor  proposals.  This  directly  relates  to  life  cycle  cost  (LCC)  because  of  the 
focus  placed  on  sustainability  cost,  typically  the  largest  component  of  LCC.  This  is 
further  discussed  in  the  Other  Documents  section  Chapter  4,  specifically  RADM  (R) 
Don  Eaton’s  published  perspective  that  poor  reliability  estimates  distort  true 
sustainment  costs  and  that  poor  reliability  is  a  large  cost  driver.  If  the  contention  is 
accurate  that  poor  reliability  estimates  are  a  major  cost  driver,  this  should  soon 
begin  to  appear  in  the  root  cause  analyses  that  are  mandated  for  programs  that 
experience  significant  cost  overruns. 

Director  DT&E  and  Director  of  Systems  Engineering  are  required  by  WSARA 
to  issue  guidance  on  and  detailed  measureable  performance  criteria  related  to 
Systems  Engineering  Master  Plans  (SEMPs)  and  Developmental  Test  and 
Evaluation  (DT&E)  plans  for  MDAPS.  Among  these  measureable  performance 
criteria  would  likely  be  reliability,  availability,  maintainability,  and  related  O&S  costs; 
WSARA  mandates  establishment  of  a  database  to  record  and  track  weapon  system 
performance  data  (WSARA,  2009,  §  102). 


JROC.  The  WSARA  specifically  charges  the  SECDEF  to  ensure  that  the 
JROC  is  engaged  in  consideration  of  trade-offs  among  cost,  schedule,  and 
performance  objectives  (§  201 ).  It  was  noted  in  our  2003  R-TOC  report  that  the 
JROC  was  not  focused  on  TOC  and  that  the  leadership  was  not  “speaking  with  one 
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voice”  concerning  the  importance  of  TOC  (Boudreau  &  Naegle,  2003,  p.  49).  This 
now  appears  to  have  been  addressed  as  a  matter  of  law. 


Milestone  Decision  Authority  (MPA).  The  WSARA  mandates  that  MDA 
ensure  appropriate  trade-offs  among  cost,  schedule,  and  performance  objectives  to 
increase  confidence  that  the  program  is  affordable  (WSARA,  2009,  §  201).  The 
words  are  straightforward  and  unambiguous,  but  the  interpretation  of  the  “cost” 
element  must  be  correctly  applied  to  system  life  cycle  cost,  not  procurement  cost. 


Competition  throughout  the  life  cycle  (WSARA,  2009,  §  202).  The  WSARA 
identifies  10  different  approaches  that  may  be  incorporated  into  a  MDAP  acquisition 
strategy  to  ensure  competition  be  used  if  cost  effective.  The  list  includes  competitive 
prototyping;  dual-sourcing;  unbundling  of  contracts;  use  of  modular,  open 
architecture  to  enable  competition  for  upgrades;  use  of  build-to-print  approaches; 
and  acquisition  of  complete  TDPs — along  with  several  other  approaches.  These 
suggested  measures  involve  competition  among  prime  contractors  and  also  among 
subcontractors  at  such  tiers  as  appropriate.  The  WSARA  views  competition  as 
extending  into  operations  and  sustainment  of  MDAPs. 


Competitive  Prototyping  (WSARA,  §  203).  The  WSARA  mandates  that 
MDAPs  include  competitive  prototyping  in  their  acquisition  strategies  prior  to  MS  B 
approval — that  is,  during  Technology  Development  Phase — unless  waived  by  the 
Milestone  Decision  Authority.  Waivers  are  specifically  limited  to  cost  effectiveness 
or  failure  to  meet  critical  national  security  objectives.  In  the  event  of  a  waiver  of 
competitive  prototyping  because  it  is  not  cost  effective,  non  competitive  prototyping 
of  the  system  is  still  required  prior  to  MS  B,  if  benefits  exceed  cost  and  are 
consistent  with  national  security  objectives  (WSARA,  2009,  §  203).  The  importance 
of  competitive  prototyping  is  that  contractors  will  feel  competitive  pressure  earlier  in 
the  process.  While  generally  considered  good  for  the  taxpayer,  it  is  arguable  that 
competitive  prototyping  has  not  resulted  in  cost  reduction  for  the  Joint  Strike  Fighter 
(JSF),  which  engaged  in  competitive  prototypes  but  afterward  experienced 
significant  cost  growth.  While  there  may  be  multiple  reasons  for  the  cost  growth 
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experienced,  the  JSF  program  may  reflect  contractor  “buy-in,”  long  a  problem  in 
DoD  acquisition  programs. 


Milestone  Decision  Authority  Certifications  and  Follow-on  Notifications 
(WSARA,  2009,  §  204).  Title  10  U.S.C.  2366a  has  been  tightened,  requiring  in  the 
MDA’s  Milestone  A  certifications  that  Congress  be  notified  if  a  program  experiences 
or  anticipates  a  cost  slip  of  25%  or  anticipates  a  schedule  slip  of  25%  or  more  in 
meeting  Initial  Operational  Capability  (IOC).  The  MDA  shall  notify  Congress  within 
30  days  of  identifying  root  causes  and  appropriate  performance  measures  to  guide 
the  rest  of  the  program  development,  and  specifically  addressing  (a)  the  essentiality 
of  the  program  to  national  security,  (b)  the  lack  of  less  costly  alternatives,  (c)  new 
estimates  of  reasonable  cost  and  schedule  for  the  program,  and  (d)  the  adequacy  of 
the  program’s  management  structure  to  control  program  cost  and  schedule. 

Milestone  B  Certification  Modification  (WSARA,  2009,  §  205).  For  programs 
that  have  gone  through  Milestone  B  decision,  10  U.S.C.  2366b  certification  has  been 
amended  by  WSARA  to  require  that  Congress  be  notified  of  waivers  by  the  MDA,  in 
writing  within  30  days,  explaining  the  basis  for  a  waiver.  The  MDA  must  review  a 
troubled  program  at  least  annually  to  determine  the  extent  that  the  program  is 
satisfying  the  certification  terms,  until  such  time  as  the  MDA  determines  that  the 
program  has  satisfied  all  the  elements  of  the  certification.  Budget  documents 
submitted  to  Congress  or  the  President  must  be  clearly  marked  as  not  fully  satisfying 
the  certification  until  the  program  has  made  the  necessary  corrections. 


Nunn-McCurdv  Cost  Breach  Modifications  (WSARA,  2009,  §  206.  Title  10 
U.S.C.  2433  (generally  known  as  the  Nunn-McCurdy  Cost  Breaches)  has  been 
amended  by  WSARA  to  require  that  the  MDA  consult  with  the  JROC  regarding 
program  requirements.  Then  the  MDA  must  determine  the  root  cause  or  causes  of 
the  cost  growth  and,  in  consultation  with  Director  CAPE,  assess  the  following:  the 
cost  of  completing  the  program  with  and  without  reasonable  modification,  the  rough 
order  of  magnitude  of  proceeding  with  an  alternative  system  or  capability,  and  the 
need  to  shift  funds  from  other  programs  due  to  the  cost  growth.  There  is  a 
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presumption  of  program  termination  unless  the  MDA  notifies  Congress  of  a  waiver 
within  60  days.  If  the  program  is  not  terminated,  the  Secretary  shall  restructure  the 
program,  rescind  the  most  recent  Milestone  approval,  require  a  new  Milestone 
approval,  and  require  onerous  additional  program  reviews). 

The  WSARA  of  2009  Summary.  There  is  no  doubt  that  the  demands  made  in 
WSARA  increased  the  rigor  and  discipline  required  in  acquisition  and  will  be 
reflected  in  more  careful  cost  estimation,  increased  caution  in  reviewing 
technological  maturity  before  advancing  programs  to  the  next  acquisition  step  or 
phase,  better  systems  engineering  and  test  planning,  and  renewed  reliance  on 
competition.  All  of  these  facets  have  the  potential  to  better  control  life  cycle  cost. 
Conversely,  all  the  same  facets  introduce  the  potential  for  added  bureaucracy  and 
unnecessary  delay.  The  WSARA  initiatives  address  past  shortcomings  in  MDAP 
acquisitions  that  have  contributed  to  the  increase  of  LCC.  Whether  these  initiatives 
will  reduce  cost  through  better  management  or  increase  cost  through  additional 
bureaucracy  remains  to  be  seen. 

B.  National  Defense  Authorization  Act  for  Fiscal  Year  2010, 
Section  805 

The  National  Defense  Authorization  Act  for  Fiscal  Year  2010  has  special 
relevance  to  life  cycle  cost,  as  will  be  explained.  In  this  law  Congress  mandated  the 
Product  Support  Manager  (PSM)  participation  in  MDAPs.  The  law  emphasized  that 
the  PSM  works  for  the  PM,  but  is  also  specifically  tasked  to  focus  on  product 
sustainment  (O&S)  cost.  This  law  increased  the  stature  of  the  program’s  chief 
logistician,  the  individual  who  is  responsible  to  develop  product  support  strategy  for 
the  warfighting  system.  Although  a  logistician,  the  PSM  is  responsible  for 
conducting  cost  analyses  to  validate  the  product  support  (sustainment)  strategy, 
including  cost-benefit  analyses  that  are  described  in  OMB  Circular  A-94.  The  PSM 
is  tasked  to  balance  PBL  support  for  optimization.  He  or  she  must  review  and 
revalidate  product  support  strategies  prior  to  a  change  in  strategy  or  every  five  years 
(National  Defense  Authorization  Act,  2010,  §  805).  The  congressional  conferees 
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recognized  that  product  support  encompasses  a  wide  range  of  logistics  functions, 
including  readiness,  reliability,  availability,  logistics  burden  (footprint)  reduction — all 
of  which  explicitly  or  implicitly  impact  ownership  cost  (Kobren,  2010,  p.  192).  The 
National  Defense  Authorization  Act  for  FY  2010  very  apparently  established  a 
position  within  the  MDAP  PM  office  that  is  responsible  for  sustainment  cost,  to 
include  reliability,  which  directly  influences  sustainment  cost. 

C.  Duncan  Hunter  National  Defense  Authorization  Act  for 
Fiscal  Year  2009,  Section  814,  Configuration  Steering 
Boards  for  Cost  Control  Under  Major  Defense  Acquisition 
Programs 

DoD  Instruction  5000.02  of  December  2008  very  succinctly  promulgates  the 
requirements  of  section  814  of  the  Duncan  Hunter  National  Defense  Authorization 
Act  for  FY  2009,  specific  to  the  formation  of  configuration  steering  boards  (CSBs)  as 
follows: 


The  Acquisition  Executive  of  each  DoD  Component  shall  establish  and 
chair  a  CSB  with  broad  executive  membership  including  senior 
representatives  from  the  Office  of  the  USD  (AT&L)  and  the  Joint  Staff. 
Additional  executive  members  shall  include  representatives  from  the  office  of 
the  chief  of  staff  of  the  Armed  Force  concerned,  other  Armed  Forces 
representatives  where  appropriate,  the  military  deputy  to  the  CAE  and  the 
Program  Executive  Officer  (PEO)  (section  814  of  P.L.  1 10-417,  Reference 
(w)). 

(1 )  The  CSB  shall  meet  at  least  annually  to  review  all  requirements 
changes  and  any  significant  technical  configuration  changes  for  ACAT 
I  and  IA  programs  in  development  that  have  the  potential  to  result  in 
cost  and  schedule  impacts  to  the  program.  Such  changes  will  generally 
be  rejected,  deferring  them  to  future  blocks  or  increments.  Changes 
shall  not  be  approved  unless  funds  are  identified  and  schedule  impacts 
mitigated. 

(2)  The  PM,  in  consultation  with  the  PEO,  shall,  on  a  roughly  annual  basis, 
identify  and  propose  a  set  of  descoping  [sic]  options,  with  supporting 
rationale  addressing  operational  implications,  to  the  CSB  that  reduce 
program  cost  or  moderate  requirements.  The  CSB  shall  recommend  to 
the  MDA  (if  an  ACAT  ID  or  1AM  program)  which  of  these  options 
should  be  implemented.  Final  decisions  on  descoping  [sic]  option 
implementation  shall  be  coordinated  with  the  Joint  Staff  and  military 
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department  requirements  officials.  (USD[AT&L],  2008c,  Enclosure  2,  p. 
30,  para  9.d.) 

This  law  introduces  a  strong  bias  toward  limiting  design  changes  to  systems. 
Note  the  Service  user  representative  is  not  named  as  member  of  the  CSB.  The 
presumption  may  be  that  the  user  would  tend  to  encourage  requirements  growth  and 
costly  changes.  The  CSB,  for  its  part,  will  listen  to  the  proposed  change  and  make 
the  board  recommendations  to  the  program  MDA.  In  Part  2,  the  PM  is  directed  to 
propose  de-scoping  options  to  reduce  cost  and  requirements.  The  MDA  is  required 
to  coordinate  changes  with  the  Joint  Staff  and  component  requirements  officials  (i.e., 
user  representatives).  The  wording  clearly  indicates  a  bias  against  changes  that  will 
increase  cost,  or  at  the  least  deferring  such  changes  to  a  future  block  or  increment. 
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IV. 


Relevant  Reports 


A.  GAO/T-N  SI  AD-98 -1  23  and  other  GAO  reports  on 
Knowledge  Point  Management 

Knowledge  point  management  can  be  used  to  avoid  program  delays  and  the 
additional  cost  that  accompanies  schedule  delays.  For  more  than  12  years  the  GAO 
has  advocated  the  use  of  knowledge  point  management  to  guide  development  of 
warfighting  systems  and  to  control  the  advancement  of  programs  until  said  systems 
have  demonstrated  their  readiness  to  proceed  to  the  next  step  in  the  development 
process  ( Defense  Acquisition:  Improved  Program  Outcomes,  1998).  The  three 
knowledge  points  recommended  by  the  GAO  are  described  in  the  following 
paragraphs. 

Knowledge  Point  1  occurs  near  Milestone  B.  The  user’s  requirements  must 
be  synchronized  with  technology  that  is  mature  enough  to  support  the  endeavor, 
allow  sufficient  time  scheduled  to  succeed,  and  provide  sufficient  funding  to 
complete  the  development  (GAO,  2003,  p.  16).  This  knowledge  point  became 
relatively  better  understood  at  such  time  as  the  Technology  Readiness  Level 
Deskbook  was  published  in  2005  (Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  [DUSD(S&T)],  2005).  Matching  requirements  against  resources  is 
a  matter  of  discipline,  and  having  the  requisite  knowledge  before  proceeding  on  is 
necessary  because  if  any  one  of  the  several  elements  is  absent  (such  as  the 
application  of  required  technologies  while  they  are  still  immature),  the  program  will 
likely  be  delayed  and  the  impact  on  cost  may  be  severe.  Continuing  GAO  reviews 
have  shown  that  Knowledge  Point  1  demands  enormous  discipline  that  has, 
unfortunately,  often  been  beyond  the  discipline  demonstrated  by  DoD  leadership 
over  many  years.  In  addition,  software  development  (not  reuse,  commercial  off-the- 
shelf  [COTS],  or  Government  off-the-shelf  [GOTS])  tends  to  behave  as  a  new, 
immature  technology,  with  each  effort  started  from  scratch.  To  combat  this  inherent 
problem,  potential  MDAP  and  MAIS  software  developers  must  undergo  a  maturity 
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evaluation  like  the  Software  Engineering  Institute’s  (SEI)  Capability  Maturity  Model- 
Integrated  (CMMI)  and  achieve  a  certain  level  of  maturity  through  independent 
evaluation.  For  the  Capability  Maturity  Models,  the  potential  software  developer 
must  achieve  Level  3  or  higher  to  be  eligible  to  compete  for  the  development 
(USD[AT&L,  2003],).  It  is  also  advisable  for  the  PM  office  to  be  similarly  evaluated 
using  something  like  the  SEI’s  Software  Acquisition  (SA)  CMM  to  help  minimize  the 
maturity  risk  with  developing  software.  More  often  than  not,  programs  are 
authorized  to  move  into  the  Engineering  and  Manufacturing  Development  (EMD) 
phase  before  the  technology  is  sufficiently  mature  to  support  a  detailed  design. 

Knowledge  Point  2  occurs  when  the  design  demonstrates  that  it  is  able  to 
meet  performance  requirements.  The  design  must  be  stable  (i.e.,  90%  of  the 
engineering  drawings  must  be  complete)  and  testing  must  show  that  the  system 
performs  at  an  acceptable  level  (GAO,  2003,  p.  16).  This  point  is  verified  at  the 
post-CDR  assessment.  Although  it  would  seem  that  completion  of  90%  of  the 
engineering  drawings  is  not  a  severe  metric,  this  is  quite  demanding;  failure  to  abide 
by  this  knowledge  point  is  likely  to  slow  down  prototype  build  and  result  in  prototype 
test  failures  caused  by  designs  that  were  not  quite  ready  for  prime  time. 

Knowledge  Point  3  occurs  when  the  system  can  be  manufactured  within  cost, 
schedule,  and  quality  targets  and  operates  reliably  (GAO,  2003,  p.  16).  In  statistical 
process  control  terms,  critical  manufacturing  processes  are  in  control  and 
consistently  producing  within  quality  standards  and  design  tolerances.  Reliability  is 
demonstrated  in  iterative  testing  (i.e.,  comparison  testing  of  manufactured  product). 
This  point  should  be  demonstrated  during  LRIP,  prior  to  the  FRP  decision.  Failure  to 
achieve  this  knowledge  point  will  result  in  manufacturing  delays,  high  costs  of 
reworking  or  repairing  manufacturing  defects,  and  customers  unhappy  with  the 
weapon  system  quality. 


Knowledge  Point  Management  is  not  new.  The  GAO  did  not  invent  the 
approach  in  1998.  It  borrowed  the  idea  from  industry,  recognizing  that  the  technique 
should,  and  could,  be  applied  to  DoD  acquisition.  Recent  changes  to  the  Defense 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-28  - 


Acquisition  System  have  largly  embraced  Knowledge  Point  Management.  Getting 
acquisition  programs  synchronized  with  this  approach  has  not  happened  overnight 
and  is  unlikely  to  happen  for  all  programs  if  not  strictly  enforced  by  DoD  leadership. 

Evolutionary  Acquisition.  The  use  of  evolutionary  acquisition  fits  conveniently 
with  Knowledge  Point  1 ,  discussed  previously.  Sometimes  technology  does  not 
become  mature  as  soon  as  hoped.  Depending  on  the  circumstances,  technological 
immaturity  might  delay  a  Milestone  B  decision  and  the  associated  program  new¬ 
start.  In  some  cases,  a  technology  that  matures  slower  than  needed  may  be 
substituted  by  an  alternative  technology  that  is  mature  and  immediately  available. 
Plainly,  this  hinges  on  whether  or  not  the  developing  system  can  result  in  an 
increment  of  useful  warfighting  capability — as  determined  by  the  sponsor/user. 

Even  when  this  happens,  the  program  faces  a  difficult  path  that  requires  “extra” 
milestones  that  are  exhausting  to  a  program  office  staff.  Such  is  the  nature  of 
evolutionary  acquisition — avoiding  one  dilemma  and  replacing  it  with  another.  The 
evolutionary  approach  places  heavy  demands  on  a  program  office,  which  must 
prepare  for  a  series  of  otherwise  unnecessary  milestones.  Is  it  worth  it? 

The  temptation  might  be  to  move  the  program  ahead,  betting  that  the  required 
technology  solution  will  miraculously  arrive  or  mature  just  in  time.  While  miracles 
sometimes  happen,  they  should  not  be  the  anticipated  substitute  for  a  sound,  well- 
planned,  and  executable  strategy. 

The  logistics  impact  cannot  be  ignored,  either.  The  result  will  either  be 
multiple  configurations  or  an  expensive  modification/upgrade  program.  Such 
impacts  might  play  out  for  many  years  or  even  for  the  lifetime  of  the  warfighting 
system.  This  may  be  associated  training  issues,  repair  parts  configuration  issues, 
software  patches,  and  operational  impacts.  The  cost  of  evolutionary  acquisition 
could  conceivably  approach  or  even  exceed  the  original  cost  of  the  program  delay. 
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The  right  answer  in  acquisition  depends  on  the  circumstances.  The  effect  on 
ownership  cost  should  always  be  one  of  the  metrics  used  to  select  the  best  course 
of  action. 

B.  GAO  Report  10-717 

In  July  2010,  the  GAO  published  Defense  Management:  DOD  Needs  Better 
Information  and  Guidance  to  More  Effectively  Manage  and  Reduce  Operating  and 
Support  Costs  of  Major  Weapon  Systems  (GAO  10-717,  2010b).  This  report  painted 
a  dreary  picture  of  relevant  databases.  The  GAO  found  that  important  O&S  cost- 
estimate  documents  had  not  been  retained  and  that  there  were  apparent  gaps  in  the 
DoD’s  ability  to  capture  actual  O&S  costs  through  the  Services’  Visibility  and 
Maintenance  of  Operations  and  Support  Costs  databases  (VAMOSC;  GAO,  2010b, 
p.  16).  Data  in  VAMOSC  and  other  Service  information  systems  or  sources  was 
inaccurate  and  incomplete  (GAO,  2010b,  pp.  16-20).  The  report  stated  that  the 
important  MDAP  system  life  cycle  cost  estimates  were  not  being  routinely  retained 
or  updated,  nor  was  there  policy  requiring  that  this  be  done.  The  GAO  pointed  out 
that  there  were  no  agreed  to  O&S  cost  elements  or  metrics  for  tracking  and 
assessing  actual  O&S  cost  performance  for  the  various  categories  of  weapon 
systems,  but  it  noted  that  the  Services  should  be  required  to  collect  and  assess  such 
data  and  maintain  it  in  their  VAMOSC  databases.  GAO  singled  out  the  Army  in 
particular  as  needing  to  develop  and  implement  a  strategy  for  improving  its 
VAMOSC  system.  On  August  19,  2010,  Director  CAPE  was  quoted  by  Inside  the 
Pentagon  as  asking  for  relief  from  the  requirement  to  establish  O&S  baselines  for 
warfighting  systems,  ostensibly  reporting  that  this  initiative  would  be  “infeasible  and 
not  advisable”  (Mishory,  2010)  because  the  VAMOSC  database  was  severely 
flawed.  Director  CAPE  appears  to  be  in  agreement  with  the  GAO  that  sustainment 
data  is  flawed  or  missing  and  not  suitable  for  rigorous  analysis  and  assessment.  A 
review  of  the  GAO  report  suggests  an  array  of  difficulties  in  comparing  actual  data  to 
baselines.  Aircraft  systems  reviewed  by  GAO  appeared  to  cost  less  than  expected 
because  the  quantities  operating  in  the  fleet  were  generally  different  (usually  fewer) 
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than  expected  (GAO,  2010b,  p.  21).  However,  costs  also  were  affected  by 
unexpected  changes  in  OPTEMPO  (specifically,  flying  hours;  GAO,  2010b,  p.  22). 
Although  both  those  factors  might  upset  budget  predictions,  they  need  not  upset 
performance  predictions;  rather,  if  shown  as  “cost  per  usage,”  reasonable 
comparisons  might  show  the  weapon  system’s  performance  against  baseline 
performance.  Cost  per  mile  or  cost  per  flying  hour  or  round  fired  could  be  compared 
to  early  cost  estimates,  as-tested  costs,  and  changes  in  cost  per  year.  Such 
comparisons  would  never  be  perfect,  but  they  would  suggest  whether  a  weapon 
system  was  performing  within  the  expected  range. 

VAMOSC  data,  by  its  very  nature,  is  collected  from  many  and  varied 
locations — sometimes  in  garrison  or  home  port,  sometimes  in  operational  and 
combat  areas.  There  should  never  be  an  expectation  of  perfect  or  highly  refined 
data,  but,  outside  of  combat  areas,  data  ought  to  be  collected  that  is  “good  enough” 
to  support  assessments  as  to  whether  equipment  is  operating  in  the  expected 
performance  range,  if  metrics  are  established  for  expected  cost  drivers.  As 
suggested  in  our  2003  report,  sample  data  collection  (SDC),  although  expensive, 
provides  a  method  for  improving  the  accuracy  of  logistics  and  O&S  cost  data. 

Looking  specifically  at  aviation  systems  across  the  Services,  the  GAO 
reported  that  most  systems  had  no  record  of  O&S  cost  estimates  related  to  key 
milestone  decisions.  Two  aircraft  systems,  the  Air  Force  F-22A  fighter  and  the  Navy 
F-A  18F/G,  did  have  some  recorded  O&S  cost  estimates  (GAO,  2010b,  pp.  24-26). 
The  two  cited  examples  suggest  the  seriousness  of  O&S  cost-estimating  inaccuracy 
and/or  cost  growth.  F-22A  actual  cost  per  flight  hour  in  2007  was  $55,783 — 67% 
higher  than  the  $33,762  that  had  been  projected  in  the  2007  President’s  Budget. 
Similarly,  on  a  flight  hour  basis,  the  Navy  F-A  18E/F  cost  $15,346  per  flight  hour  of 
operation — 40%  higher  than  the  $10,979  predicted  in  1999. 
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C.  GAO-08 -1 1  59T,  Defense  Acquisitions:  Fundamental 
Changes  are  Needed  to  improve  Weapon  Procurement 
Outcomes 

In  his  testimony,  the  GAO  Director  of  Acquisition  and  Sourcing  Management, 
Michael  Sullivan,  succinctly  identified  systemic  problems  that  led  to  poor  acquisition 
outcomes  (GAO,  2008).  His  findings  identified  disconnects  in  the  three  systems  that 
are  essential  to  the  acquisition  of  military  weapons — Planning,  Programming, 
Budgeting,  and  Execution  process  (PPBE),  Joint  Capabilities  Integration  and 
Development  System  (JCIDS),  and  the  Defense  Acquisition  System.  He  further 
characterized  a  breakdown  of  systems  engineering  at  critical  junctures,  referred  to 
as  knowledge  points.  He  also  described  a  culture  in  the  Services  and  the  DoD  that 
incentivize  overpromising  system  performance  and  underestimating  cost  and 
schedule,  a  pervasive  problem  across  the  DoD  for  many  years. 

D.  DoD  Weapon  System  Acquisition  Reform  Product  Support 
Assessment 

In  the  Product  Support  Assessment  Team’s  (PSAT)  November  2009  report 
DoD  Weapon  System  Acquisition  Reform  Product  Support  Assessment,  they  listed 
eight  areas  to  improve  product  support.  At  least  five  of  the  areas  impact  system  life 
cycle  cost  (e.g.,  Product  Support  Business  Model,  Metrics,  Operating  and  Support 
Costs,  Analytical  Tools,  and  Human  Capital;  PSAT,  2009,  pp.  12-13). 

E.  Institute  for  Defense  Analyses  Study:  The  Major  Causes  of 
Cost  Growth  in  Defense  Acquisition 

The  2009  Institute  for  Defense  Analyses  (IDA)  study,  led  by  Gene  Porter, 
examined  1 1  MDAP  systems  that  had  exhibited  significant  cost  growth  between 
1995  and  2006.  The  primary  causes  of  cost  growth  stemmed  from  two  defects: 
“Weaknesses  in  management  visibility,  direction,  and  oversight”  and  “Weaknesses 
in  initial  program  definition  and  costing,”  neither  of  which  was  a  new  phenomenon 
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(Porter  et  al. ,  2009,  pp.  ES-6 — ES-14).  Much  of  the  blame  for  the  first  weakness 
was  “a  general  lack  of  discipline”  (Porter  et  al.,  2009,  p.  ES-6). 

Porter  et  al.  (2009)  make  a  series  of  recommendations  that  are  intended  to 
address  the  causes  of  cost  growth  reflected  in  their  study;  their  recommendations 
are  supportive  of  the  goals  of  WSARA  of  2009  (Porter  et  al.,  2009,  pp.  ES-15-ES- 
18). 

F.  Other  Documents 

In  his  memorandum,  State  of  Reliability,  Dr.  J.  Michael  Gilmore,  the  Director 
of  Operational  Test  and  Evaluation  (DOT&E,  2010),  made  the  link  that  poor  reliability 
is  a  major  contributor  to  LCC.  The  implication  is  that  the  long-held  28-72  LCC 
statistics  could  be  altered  by  front-end  attention  to  reliability  growth.  That  is, 
investing  more  RDTE  funding  in  reliability  improvement  at  the  front  end  could  result 
in  higher  reliability  components  that  would  cost  less  to  operate,  malfunctioning  less 
often.  The  remarkable  thing  here  is  that  program  leadership  has  tried  to  improve 
reliability  in  many,  if  not  all,  programs.  Dr.  Gilmore  made  reference  to  a  recently 
published  reliability  standard,  ANSI/GEIA-STD-0009,  which  should  be  employed. 

He  quoted  a  May  2008  Defense  Science  Board  (DSB)  report,  which  stated  that  “high 
suitability  (reliability)  failure  rates  were  caused  by  the  lack  of  a  disciplined  systems 
engineering  process,  including  a  robust  reliability  growth  program”  (DSB,  2008  in  the 
Task  force  Chairman’s  cover  letter).  The  DSB  further  emphasized  that  the  “single 
most  important  step... is  to... execute  a  viable  systems  engineering  strategy  from  the 
beginning,  including  a  robust  reliability,  availability,  and  maintainability  (RAM) 
program”  (DOT&E,  2010,  pp.1-2;  DSB,  2008  p.23). 

Dr  Gilmore  made  his  case  further  by  stating, 

I  understand  that  directing  use  of  ANSI/GEIA  STD-0009  is  a  change  from 

business  as  usual.  That  change  is  urgently  needed.  Requiring  the  use  of 

0009  is  appropriate  for  the  following  reasons: 
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■  0009  is  credible.  To  obtain  an  ANSI  certification,  0009  was  peer 
reviewed  by  350  subject  matter  experts  (SMEs)  from  all  walks  of  the 
reliability  community,  including  government,  Services,  academia,  and 
industry. 

■  0009  is  new,  different,  necessary.  ANSI/GEIA  STD-0009  is  not  similar 
to  MIL-STD-785B.  The  two  standards  are  quite  different,  and  MIL-STD- 
785B  will  not  suffice.  MIL-STD-785B  required  a  “level-of-effort”  and 
discrete  tasks,  but  not  system  engineering  processes.  MIL-STD-785B 
had  no  systematic  processes  to  identify  and  mitigate  failure  modes 
throughout  the  product  life  cycle.  0009  corrects  the  failings  of  785B. 

■  0009  has  become  a  model  for  others.  Since  publication  of  ANSI/GEIA 
STD-0009,  major  standards  such  as  SAE  JA  1000  and  IEEE  1332  are 
now  being  rewritten  to  embrace  the  science-based,  closed-looped 
approach  of  ANSI/GEIA  STD-0009. 

■  0009  has  been  formally  adopted  by  DoD  for  use  (August  20,  2009). 
ANSI/GEIA  STD-0009  will  ensure  a  systems  level  approach  to  identify 
and  mitigate  failure  modes  until  requirements  are  met.  (DOT&E,  2010, 
p.  3) 

In  his  own  words,  Dr.  Gilmore  has  publically  entered  into  the  reliability  dialog 


because 


discussions  that  have  occurred  among  our  staffs  participating  in  the  re¬ 
convened  Reliability  Working  Group  indicate  that  there  is  some  question  as  to 
whether  reliability  is  an  important  issue,  and  there  also  appear  to  be 
questions  about  the  merits  of  the  reliability  standard  ANSI/GEIA-STD-0009. 
(DOT&E,  2010,  p.  1) 

Dr.  Gilmore  emphatically  stated  in  the  very  next  paragraph  of  his  memo  that 
there  is  no  question  about  it.  That  is,  defense  acquisition  systems  completing  R&D 
are  often  not  reliable,  and  he  linked  poor  reliability  to  sustainment  costs  that  are 
higher  than  necessary  (DOT&E,  2010,  p.  1).  This  is  reflective  of  findings  in  RADM 
(R)  Don  Eaton’s  2004  paper,  discussed  below: 

RADM  Don  Eaton,  retired  Arthur  Chair  in  Logistic  Management  at  the  Naval 
Postgraduate  School,  said  in  a  July  24,  2010,  e-mail,  “If  we  thoughtfully  analyzed  the 
FOMs  [figures  of  merit]  of  COST,  SCHEDULE  AND  PERFORMANCE  we  would 
always  conclude  poor  reliability  is  THE  dominant  cost  driver  as  well  as  a  key  player 
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in  mission  failure.”  In  his  August  2004  paper  Improving  the  Management  of 
Reliability,  he  provided  a  stunning  example  from  naval  aviation,  the  trailing  edge  flap 
actuator  for  the  F/A-18  A-D.  He  pointed  out  that  the  component  reliability  was  set  at 
4,000  hours  mean  time  between  failure  (MTBF).  In  operation,  the  demonstrated 
performance  in  MTBF  was  138  hours,  3.45%  of  what  it  was  supposed  to  be  (Eaton, 
2004,  pp.  5-6).  RADM  Eaton  did  not  attempt  to  calculate  the  impact  to  sustainment 
cost  because  that  was  not  the  purpose  of  his  paper.  Nevertheless,  without 
calculating  the  impact  in  dollars,  one  can  see  that  such  poor  performance  reflects  in 
significantly  increased  costs  in  maintenance  man-hours  for  repair,  repair  parts 
consumed,  transportation  of  repair  parts  and/or  replacement  components,  and 
required  stockage  levels  that  had  to  be  maintained,  not  to  mention  the  impact  on  the 
aircraft’s  mission  availability  rate.  Such  examples  are  not  unique  to  aircraft,  or  to  the 
Navy.  Many,  if  not  all,  programs  have  reliability  “bad  actors”  that  need  to  be 
redesigned  and  replaced  because  of  what  they  are  costing  in  maintenance  time, 
repair  parts  expense,  and  transportation.  This  situation  could  be  improved  by 
rigorous  reliability  improvement  programs  during  system  development,  as  described 
in  the  statements  by  Dr.  Gilmore  referred  to  previously.  This  would  require 
disciplined  leadership,  PMOs  determined  to  get  the  design  right,  and  user  insistence 
that  reliability  goals  are  set — and  achieved — for  warfighting  systems. 

Collaborative  Tools.  Reliability  improvement  is  bolstered  by  the  involvement 
of  product  support  managers  as  encouraged  in  the  National  Defense  Authorization 
Act  for  Fiscal  Year  2010,  Section  805.  The  reliability  improvement  process  can  be 
enhanced  by  the  use  of  collaborative  tools  to  involve  life  cycle  logistics  professionals 
and  make  available  repair  parts  databases  to  sharpen  design  decisions.  This  effort 
can  be  further  helped  by  Pareto  analysis — that  is,  focus  on  the  cost  drivers,  primarily 
the  expensive  items  that  break  more  often  than  predicted.  This  approach  can  be 
used  early  in  the  design  process,  too,  by  searching  systems  command  and  DLA 
databases  to  examine  performance  of  similar  or  predecessor  systems. 
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Wider  View  of  the  Need  for  O&S  Cost  Databases.  It  is  easy  for  field  users, 
maintainers,  and  PMs  to  visualize  cost  databases  that  can  be  used  to  identify  cost 
drivers  in  fielded,  legacy  systems.  This  is  important  work  and  a  principal  focus  of 
VAMOSC  databases,  maintained  by  each  of  the  Services.  However,  it  must  also  be 
recognized  that  O&S  databases  are  needed  to  support  early  O&S  calculations  of 
emerging  systems,  still  in  pre-acquisition.  In  her  2010  report,  Marti  A.  Roper 
discussed  the  need  for  databases  that  support  acquisition  cost  estimates — down  to 
subsystem  or  component  levels,  showing  cost  ranges.  Such  a  knowledge  base  is 
critical  for  the  development  of  follow-on  systems  so  that  known  cost  drivers  can  be 
addressed  for  potentially  significant  life  cycle  cost  savings  with  deployment  of  the 
replacement  system.  Roper  referred  to  this  as  capabilities-based  parametric  data 
analysis  (2010,  pp.  71-73). 
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V. 


Policy  Pronouncements 


The  OSD  implemented  the  2009  version  of  WSARA  on  December  4,  2010, 
through  the  USD(AT&L)  publication  of  Directive  Type  Memorandum  (DTM)  09-027 
(USD[AT&L],  2009).  About  10  months  later,  on  October  21,  2010,  the  USD(AT&L) 
amended  the  original  document,  establishing  a  date  by  which  the  DoDI  5000.02  had 
to  be  revised  (USD[AT&L],  2010). 

A.  Target  Affordability  and  Control  Cost  Growth  for  ACAT  I 
Programs 

Corollary  to  WSARA  implementation,  the  USD(AT&L)  published  the 
Implementation  Directive  for  Better  Buying  Power— Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending  (USD[AT&L],  201  Od).  The  intent  of  this 
implementation  directive  was  to  reach  beyond  WSARA  mandates  to  obtain  greater 
affordability-based  decision-making  in  warfighting  system  programs.  Pertinent 
specifics  are  as  follows. 

•  Mandate  affordability  as  a  requirement.  PMs  are  now  required  to  treat 
affordability  like  a  Key  Performance  Parameter  (KPP)  at  Milestone  A. 
The  affordability  target  is  to  be  stated  in  two  metrics:  average  unit 
acquisition  cost  and  average  annual  operating  and  support  cost  per 
unit.  These  metrics  will  be  the  basis  for  pre-Milestone  B  decision¬ 
making  and  systems  engineering  tradeoff  analysis  to  establish  cost 
and  schedule  trade  space.  Such  a  mandate  requires  a  Database 
similar  to  the  one  Roper  described  (2010,  pp.  71-73).  This  will  provide 
a  basis  for  comparison  against  the  applicable  portfolio  or  mission  area, 
and  will  reflect  acquisition  and  O&S  budget  suitability  to  absorb  the 
proposed  program  new  start.  Analysis  must  address  specific 
adjustments  to  fit  new  programs  affordably  into  their  portfolio  or 
mission  area  (USD[AT&L],  201  Od,  p.  1). 
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The  Milestone  B  acquisition  decision  memorandum  will  include  an 
affordability  requirement  for  acquisition  cost  and  O&S  cost  that  will  be 
the  functional  equivalent  to  a  KPP  and  will  be  established  as 
Acquisition  Program  Baseline  (APB)  metrics  (USD[AT&L],  201  Od,  p.  2). 

Productivity  growth  through  will  cost/should  cost.  Should-cost  targets 
will  be  set  for  all  ACAT  I,  II,  and  III  programs  under  consideration  for 
major  milestone  decisions.  Should-cost  targets  will  be  based  on 
thoroughly  scrubbed  bottom-up  assessments,  assuming  reasonable 
efficiency  and  productivity  enhancement  effort.  Should-cost  will  be 
used  as  the  basis  of  contract  negotiation  and  incentives  to  track 
contractor  and  PEO/PM  performance  annually  (USD[AT&L],  201  Od,  p. 
2).  Independent  cost  estimates  will  establish  “forecasts  of  what  a 
program  will  cost  based  on  reasonable  extrapolations  from  historical 
experience — to  support  budgeting  and  programming”  (USD[AT&L], 
2010a,  p.  3).  The  motivation  for  industry  is  higher  profit  for  better 
performance. 

Eliminate  redundancy  within  warfighter  portfolios.  The  DoD  and  the 
components  have  begun  portfolio  reviews  to  identify  and  eliminate 
system  redundancy  in  warfighting  systems.  This  function  will  be 
accomplished  annually  by  the  military  departments  and  agencies 
(USD[AT&L],  201  Od,  p.  2). 

Make  production  rates  economical  and  stable.  This  element  is 
intended  to  synchronize  production  to  portfolio  affordability  targets  set 
at  MS  A,  as  adjusted  at  MS  B,  and  economic  order  quantity  (EOQ). 
Production  rates  will  be  part  of  the  affordability  analysis  at  MS  A  and 
MS  B.  MS  C  now  requires  a  range  of  production  rates,  and  deviation 
from  that  range  without  prior  approval  will  lead  to  revocation  of  the 
milestone  (USD[AT&L],  2010a,  p.  4). 
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•  Set  shorter  program  timelines.  Schedule  slips  are  very  expensive  and 
delay  the  arrival  of  needed  equipment  into  the  hands  of  warfighters. 
Unfortunately,  long  developmental  programs  have  been  the  norm  for 
many  years  USD[AT&L],  2010a,  pp.  4-5).  For  future  programs,  the 
program  schedule  will  be  set  at  MS  B,  consistent  with  the  cost  tradeoff 
analysis.  This  is  logical  because  cost  and  schedule  must  be 
synchronized  to  meet  affordability  targets.  Deviation  from  schedule 
without  prior  approval  will  lead  to  revocation  of  the  milestone 
(USD[AT&L],  20 1 0d ,  p.  2). 

B.  Promote  Real  Competition 

Present  a  competitive  acquisition  strategy  at  each  Milestone.  ACAT  I,  II,  III, 
and  IV  are  all  required  to  include  a  competitive  strategy  prior  to  each  milestone  and 
to  include  reduction  of  single-bid  competitions.  The  strategy  will  include  discussion 
of  market  research,  restricted  specifications,  and  adequate  time  for  proposal 
preparation.  A  2%  improvement  goal  of  one-bid  statistics  has  been  established  for 
2011  (USD[AT&L],  201  Od,  p.  4). 

Remove  Obstacles  to  Competition.  Contract  officers  are  required  to  conduct 
negotiations  with  all  single-bid  offerors,  unless  waived  by  the  Head  of  Contracting 
agency  (HCA),  and  the  basis  of  negotiation  shall  be  cost  or  price  analysis,  using 
non-certified  data.  Component  or  agency  competition  advocates  are  required  to 
achieve  an  improvement  rate  of  10%  per  year  in  effective  competition  (USD[AT&L], 
2010d,  p.  4). 

Require  open  systems  architecture  and  acquisition  of  tech  data  rights.  Use  of 
open  system  architecture  and  tech  data  rights  will  both  be  pursued  to  ensure  the 
programs’  lifetime  consideration  of  competition.  The  results  of  these  initiatives  will 
be  reported  in  the  Acquisition  Strategy  Reports  (USD[AT&L],  201  Od,  pp.  4-5).  . 
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In  summary,  the  several  USD(AT&L)  initiatives  cited  previously  from  the 
Implementation  Directive  for  Better  Buying  Power— Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending  and  from  the  Memorandum  for  Acquisition 
Professionals  entitled  Better  Buying  Power:  Guidance  for  Obtaining  Greater 
Efficiency  and  Productivity  in  Defense  Spending  specifically  addressed 
improvements  that  may  be  made  through  the  diligent  efforts  of  PMs,  contracting 
officers,  and  trained  acquisition  professionals. 


C.  Software  Acquisition  Process  Improvement  Programs 

On  March  21 , 2003,  the  OSD  issued  a  memorandum  to  the  secretaries  of  the 
military  departments  and  other  selected  recipients,  establishing  the  DoD’s  Software 
Acquisition  Process  Improvement  Program  and  directing  each  Service  to  establish  a 
similar  program  (OSD,  2003). 

While  clearly  focused  on  the  software  acquisition  process,  this  memorandum 
established  the  need  for  a  more  systemic  approach  that  would  include  requirements 
development,  configuration  management,  risk  management,  and  test  and 
evaluation,  as  well  as  all  relevant  stakeholders.  These  are  all  key  tenets  in 
designing  systems  with  desirable  TOC  characteristics,  and  including  logisticians  as 
relevant  stakeholders  is  necessary  to  help  ensure  that  the  Post  Deployment 
Software  Support  (PDSS)  planning  produces  a  robust  and  supportable  software 
architecture.  As  with  any  other  system  component,  the  software  design  architecture 
will  determine  the  supportability  performance  that  helps  drive  the  system’s  TOC. 

D.  A  Specific  Navy  Initiative:  Gate  Reviews 

The  Navy  has  instituted  a  series  of  reviews,  termed  “gate  reviews”  to  better 
control  program  development  cost.  The  Navy  Total  Ownership  Cost  Guidebook 
(Department  of  the  Navy  [DoN],  2010;  published  concurrently  with  SECNAVINST 
5000. 2E)  depicts  a  series  of  10  gate  reviews  that  stretch  across  the  pre-acquisition 
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and  acquisition  phases  and  into  the  sustainment  phase.  Each  gate  review  asks 
tailored  cost  questions  relevant  to  the  specific  life-cycle  event  (Department  of  the 
Navy,  2010,  pp.  4-32).  The  complete  array  of  gate  reviews  is  as  follows: 


■  Gate  1 — Initial  Capabilities  Document 

■  Gate  2 — Analysis  of  Alternatives 

■  Gate  3 — Capability  Development  Document 

■  Gate  4 — System  Design  Specification 

■  Gate  5 — RFP  for  Engineering  and  Manufacturing  Development 
Contract 

■  Gate  6  Reviews 

o  Integrated  Baseline  Review 
o  Post  Critical  Design  Review 
o  Capability  Production  Document 
o  Pre-Full  Rate  Production  Decision  Review 
o  Sustainment  Sufficiency  Review(s) 

At  each  gate  review,  formal  design  review,  and  assessment,  programs  must 
demonstrate  progress  toward  their  affordability  initiatives,  with  strong  consideration 
in  mitigation  or  reduction  of  TOC.  The  Navy’s  intent  is  to  change  the  culture  from 
what  the  authors  of  this  working  paper  perceive  as  a  shortsighted  goal  of  obtaining 
funds  for  development  and  procurement  to  the  more  complete  perspective  of  total 
life  cycle  cost  affordability. 


■  Gate  Review  1 ,  which  is  intended  to  shape  the  Analysis  of  Alternatives 
(AoA  )study  analysis,  requires  consideration  of  O&S  costs  based  on 
current  or  similar  systems.  AoA  study  TOC  guidance  is  intended  to  be 
sufficiently  detailed  to  inform  and  support  the  selection  of  a  materiel 
solution  from  among  the  various  AoA  alternative  candidates. 

■  Intermediate  gate  reviews  are  coupled  to  existing  systems  engineering 
and  acquisition  milestone  review  points.  These  reviews  become  a 
forum  to  assess  whether  program  tradeoffs  and  decisions  are 
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controlling  life  cycle  cost  and  whether  the  program  is  continuing  on  the 
correct  affordability  azimuth.  Each  of  the  gate  reviews  requires  briefing 
of  specific  cost  charts,  making  it  unlikely  that  cost  growth  and  schedule 
slippage  can  be  obscured. 

■  The  Gate  6  Sustainment  Review(s),  accomplished  post-IOC,  examine 
the  warfighting  system’s  actual  performance  data  compared  to  the 
system’s  KPP  thresholds  and  the  warfighting  system’s  actual  life  cycle 
cost  compared  to  its  prior  estimates  of  ownership  cost. 

In  the  aggregate,  Gate  Reviews  provide  for  oversight  and  governance  of 
MDAP  system  developments.  In  a  wider  sense,  Gate  Reviews  provide  a  forum  for 
lessons  learned  regarding  TOC  while  controlling  the  affordability  of  individual 
systems — and,  hence,  the  broader  portfolios  of  warfighting  systems — throughout  the 
developmental,  production,  and  sustainment  phases  of  warfighting  systems. 
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VI. 


Other  Initiatives 


A.  Controls  on  Software  Development 

1.  Driving  the  Software  Requirements  and  Architectures  for  System 
Supportability 

While  the  tools  and  techniques  described  in  this  section  were  designed  for  the 
software  components,  they  would  be  just  as  effective  for  any  non-software 
component  as  they  are  Systems  Engineering  (SE)  oriented.  The  SEP  focus  used 
does  not  attempt  to  separate  software  from  other  components,  so  all  system 
components  would  benefit  from  using  these  tools  and  techniques. 

a.  Software  Supportability  Analysis 

As  with  hardware  system  components,  software  supportability  attributes  must 
be  designed  into  the  system  architecture.  Many  hardware-oriented  engineering 
fields  are  now  quite  mature,  so  that  a  number  of  supportability  attributes  would  be 
automatically  included  in  any  competent  design,  even  if  they  were  not  specified  by 
the  user  community.  For  example,  the  state  of  maturity  for  the  automotive 
engineering  field  means  that,  in  any  automotive-related  program,  there  would  be 
supportability  designs  allowing  for  routine  maintenance  of  system  filters,  lubricants, 
tires,  brakes,  batteries,  and  other  normal  wear-out  items.  There  are  few,  if  any, 
corresponding  supportability  design  attributes  that  would  be  automatically  included 
in  even  the  best  software  construct.  Virtually  all  of  the  software  supportability 
attributes  required  must  be  explicitly  specified  because  they  would  not  likely  be 
included  in  the  design  architecture  without  clearly  stated  requirements.  With 
software,  you  get  what  you  specify  and  very  little  else.  So  how  does  one  ensure  that 
required  software  supportability  attributes  are  not  overlooked? 

Logistics  Supportability  Analysis  (LSA),  performed  extremely  early,  is  one  of 
the  keys  for  developing  the  system  supportability  attributes  needed  and  expected  by 
the  warfighter.  The  F/A  18  Super  Hornet  aircraft  was  designed  for  higher  reliability 
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and  improved  ease  of  maintenance  compared  to  its  predecessors  (“F/A  18,”  201 1 ) 
because  of  warfighter  needs  for  generating  combat  power  in  the  form  of  aircraft 
sorties  available.  The  LSA  performed  on  the  F/A  18  determined  that  a  design 
fostering  higher  reliability  and  faster  maintenance  turnaround  time  (the  engines  are 
attached  to  the  airframe  at  10  locations  and  can  be  changed  in  about  20  minutes  by 
a  four-man  team)  would  result  in  more  aircraft  being  available  to  the  commander 
when  needed.  The  concept  for  software  LSA  is  no  different,  but  implementing  sound 
supportability  analyses  on  the  software  components  has  been  spotty,  at  best,  and 
completely  lacking,  at  worst. 

To  assist  in  effective  software  LSA,  a  focus  on  these  elements  is  key: 
Maintainability,  Upgradeability,  Interoperability/Interfaces,  Reliability,  and  Safety  & 
Security — MUIRS. 

B.  Maintainability 

The  amount  of  elapsed  time  between  initial  fielding  and  the  first  required 
software  maintenance  action  can  probably  be  measured  in  hours,  not  days.  The 
effectiveness  and  efficiency  of  these  required  maintenance  actions  is  dependent  on 
several  factors,  but  the  software  architecture  that  was  developed  from  the 
performance  specifications  provided  is  critical.  The  DoD  must  influence  the  software 
architecture  through  the  performance  specification  process  to  minimize  the  cost  and 
time  required  to  perform  essential  maintenance  tasks. 

Maintenance  is  one  area  in  which  software  is  fundamentally  different  from 
hardware.  Software  is  one  of  the  very  few  components  in  which  we  know  that  the 
fielded  product  has  shortcomings,  and  we  field  it  anyway.  There  are  a  number  of 
reasons  why  this  happens;  for  instance,  there  is  typically  not  enough  time,  funding, 
or  resources  to  find  and  correct  every  error,  glitch,  or  bug,  and  not  all  of  these  are 
worth  the  effort  of  correcting.  Knowing  this,  there  must  be  a  sound  plan  and 
resources  immediately  available  to  quickly  correct  those  shortcomings  that  do 
surface  during  testing  and  especially  those  that  arise  during  warfighting  operations. 
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Even  when  the  system  software  is  operating  well,  changes  and  upgrades  in  other 
interfaced  hardware  and  software  systems  will  drive  some  sort  of  software 
maintenance  action  to  the  system  software.  In  other  words,  there  will  be  a 
continuous  need  for  software  maintenance  in  the  planned  complex  SoS  architecture 
envisioned  for  net-centric  warfare. 

Because  the  frequency  of  required  software  maintenance  actions  is  going  to 
be  much  higher  than  in  other  systems,  the  cost  to  perform  these  tasks  is  likely  to  be 
higher  as  well.  One  of  the  reasons  for  this  is  that  software  is  not  maintained  by 
’’maintainers,”  as  are  most  hardware  systems,  but  is  maintained  by  the  same  type  of 
people  that  originally  developed  it — software  engineers.  These  engineers  will  be 
needed  immediately  upon  fielding,  and  a  number  will  be  needed  throughout  the 
lifespan  of  the  system  to  perform  maintenance,  add  capabilities,  and  upgrade  the 
system.  There  are  several  models  available  to  estimate  the  number  of  software 
engineers  that  will  be  needed  for  support;  planning  for  funding  these  resources  must 
begin  very  early  in  the  process.  Because  the  DoD  has  a  very  limited  capability  for 
supporting  software  internally,  early  software  support  is  typically  provided  by  the 
original  developer  and  is  included  in  the  RFP  and  proposal  for  inclusion  into  the 
contract  or  as  a  follow-on  Contractor  Logistics  Support  (CLS)  contract. 

C.  Upgradeability 

A  net-centric  environment  composed  of  numerous  systems  developed  in  an 
evolutionary  acquisition  model  will  create  an  environment  of  almost  continuous 
change  as  each  system  upgrades  its  capabilities  over  time.  System  software  will 
have  to  accommodate  the  changes  and  will  have  to,  in  turn,  be  upgraded  to  leverage 
the  consistently  added  capabilities.  The  software  architecture  design  will  play  a 
major  role  in  how  effective  and  efficient  capabilities  upgrades  are  implemented,  so 
communicating  the  known,  anticipated,  and  likely  system  upgrades  will  impact  how 
the  software  developer  designs  the  software  for  known  and  unknown  upgrades. 
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Trying  to  anticipate  upgrade  requirements  for  long-lived  systems  is  extremely 
challenging  to  materiel  developers,  but  is  well  worth  their  effort.  Unanticipated 
software  changes  in  the  operational  support  phase  cost  50  to  200  times  the  cost  in 
early  design,  so  any  software  designed  to  accommodate  an  upgrade  that  is  never 
realized  costs  virtually  nothing  when  compared  to  changing  software  later  for  a 
capability  that  could  have  been  anticipated.  For  example,  the  Army  Tactical  Missile 
System  (ATACMS)  Unitary  was  a  requirement  to  modify  the  missile  from  warhead 
air  delivery  to  surface  detonation — that  is,  flying  the  warhead  to  the  ground.  The 
contract  award  for  the  modification  was  $119  million.  The  warhead  was  not  new 
technology,  nor  particularly  challenging  to  integrate  with  the  missile  body.  The  vast 
majority  of  this  cost  was  to  reengineer  the  software  to  guide  the  missile  to  the 
surface.  Had  there  been  an  upgrade  requirement  for  this  type  of  mission  in  the 
original  performance  specification,  this  original  cost  (including  potential  upgrades, 
even  if  there  were  10  other  upgrade  requirements  that  were  never  applied)  would 
have  been  a  fraction  of  this  modification  cost. 

D.  Interfaces/Interoperability 

OA  design  focuses  on  the  strict  control  of  interfaces  to  ensure  the  maximum 
flexibility  in  adding  or  changing  system  modules,  whether  they  are  hardware  or 
software  in  nature.  This  presupposes  that  the  system  modules  are  known — which 
seems  logical,  as  most  hardware  modules  are  well-defined  and  bounded  by  both 
physics  and  mature  engineering  standards.  In  sharp  contrast  to  hardware,  software 
modularity  is  not  bounded  by  physics,  and  there  are  very  few  software  industry 
standards  for  the  modular  architecture  in  software  components.  This  is  yet  another 
area  in  which  the  software  developer  needs  much  more  information  about 
operational,  maintenance,  reliability,  safety,  and  security  performance  requirements, 
as  well  as  current,  planned,  and  potential  system  upgrades.  These  requirements, 
once  well  defined  and  clearly  communicated,  will  drive  the  developer  to  design  a 
software  modular  architecture  supporting  OA  performance  goals.  For  example,  if  a 
system  uses  a  Global  Positioning  System  (GPS)  signal,  it  is  likely  that  the  GPS  will 
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change  over  the  life  of  the  system.  Knowing  this,  the  software  developer  creates  a 
corresponding  discrete  software  module  that  is  much  easier  and  less  expensive  to 
interface,  change,  and  upgrade  as  the  GPS  system  does  so. 

With  the  system  software  modular  architecture  developed,  the  focus  returns 
to  the  interfaces  between  hardware  and  software  modules,  as  well  as  to  the  external 
interfaces  needed  for  the  desired  interoperability  of  the  net-centric  force.  Software 
is,  of  course,  one  of  the  essential  enablers  for  interoperability  and  provides  a 
powerful  tool  for  interfacing  systems,  including  systems  that  were  not  designed  to 
work  together.  Software  performing  the  function  of  ’’middleware”  allows  legacy  and 
other  dissimilar  systems  to  interoperate.  Obviously,  this  interoperation  provides  a 
significant  advantage,  but  it  comes  with  a  cost  in  the  form  of  maintainability, 
resources,  and  system  complexity.  As  software  interfaces  with  other  components 
and  actually  performs  the  interface  function,  controlling  it  and  ensuring  the  interfaces 
provide  the  desired  OA  capability  becomes  a  major  software-management  and 
software-discipline  challenge. 

One  method  being  employed  by  the  DoD  attempts  to  control  the  critical 
interfaces  through  a  set  of  parameters  or  protocols  rather  than  through  active 
management  of  the  network  and  network  environment.  This  method  falls  short  on 
several  levels.  It  fails  to  understand  and  control  the  effects  of  aggregating  all  of  the 
systems  in  a  net-centric  scheme.  For  instance,  each  individual  system  may  meet  all 
protocols  for  bandwidth,  but  when  all  systems  are  engaged  on  the  network,  all 
bandwidth  requirements  are  aggregated  on  the  network — overloading  the  total 
bandwidth  available  for  all  systems.  In  addition,  members  of  the  Software 
Engineering  Institute  (SEI)  noted, 

While  these  standards  may  present  a  step  in  the  right  direction,  they  are 
limited  in  the  extent  to  which  they  facilitate  interoperability.  At  best,  they 
define  a  minimal  infrastructure  that  consists  of  products  and  other  standards 
on  which  systems  can  be  based.  They  do  not  define  the  common  message 
semantics,  operational  protocols,  and  system  execution  scenarios  that  are 
needed  for  interoperation.  They  should  not  be  considered  system 
architectures.  For  example,  the  C4ISR  domain-specific  information  (within 
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the  JTA)  identifies  acceptable  standards  for  fiber  channels  and  radio 
transmission  interfaces,  but  does  not  specify  the  common  semantics  of 
messages  to  be  communicated  between  C4ISR  systems,  nor  does  it  define 
an  architecture  for  a  specific  C4ISR  system  or  set  of  systems.  (Morris  et  al. , 
2004,  p.  38) 

Clearly,  understanding  and  controlling  the  interfaces  is  critical  for  effective 
interoperation  at  both  the  system  and  SoS  levels.  The  individual  PM  must  actively 
manage  all  systems’  interfaces  impacting  OA  performance,  and  a  network  PM  must 
do  the  same  for  the  critical  network  interfaces.  Due  to  this  necessity  of  constant 
management,  a  parameters-and-protocols  approach  to  net-centric  OA  performance 
is  unlikely  to  produce  the  capabilities  and  functionality  expected  by  the  warfighter. 

Understanding  the  software  interfaces  begins  with  the  software  architecture; 
controlling  the  interfaces  is  a  unique  challenge  encompassing  the  need  to  integrate 
legacy  and  dissimilar  systems  and  the  lack  of  software  interface  standards  within  the 
existing  software  engineering  environment.  As  stated  earlier,  the  architecture  needs 
to  be  driven  through  detailed  performance  specifications,  which  will  help  define  the 
interfaces  to  be  controlled.  An  effective  method  for  controlling  the  interfaces  is  to 
intensely  manage  a  well-defined  Interface  Control  Document  (ICD),  which  should  be 
a  Contract  Data  Requirements  List  (CDRL)  deliverable  on  any  software-intensive  or 
networked  system. 

E.  Reliability 

While  the  need  for  highly  reliable  weapon  systems  is  obvious,  the  impact  on 
total  system  reliability  of  integrating  complex  software  components  is  not  so  obvious. 
Typically,  as  system  complexity  increases,  maintaining  system  reliability  becomes 
more  of  a  challenge.  Add  the  complexity  of  effectively  networking  an  SoS  (all  of 
which  are  individually  complex)  to  a  critical  warfighting  capability  that  is  constantly 
evolving  over  time,  and  reliability  becomes  daunting. 

Once  again,  the  software  developer  must  have  an  understanding  of  reliability 
requirements  before  crafting  the  software  architecture  and  developing  the  software 
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applications.  Highly  reliable  systems  often  require  redundant  capability,  and  this 
holds  true  for  software  components  as  well.  In  addition,  software  problems  tend  to 
propagate,  resulting  in  a  degradation  of  system  reliability  over  time.  For  example,  a 
Malaysian  Airlines  Boeing  777  suffered  several  flight  control  problems  resulting  in 
the  following:  a  near  stall  situation,  contradicting  instrument  indications,  false 
warnings,  and  difficulty  controlling  the  aircraft  in  both  autopilot  and  manual  flight 
modes.  The  problems  were  traced  to  software  in  an  air  data  inertial  reference  unit 
that  was  feeding  erroneous  data  to  the  aircraft’s  primary  flight  computer  (PFC), 
which  is  used  in  both  autopilot  and  manual  flight  modes.  The  PFC  continued  to  try 
to  correct  for  the  erroneous  data  received,  adjusting  flight  control  surfaces  in  all 
modes  of  flight,  displaying  indications  that  the  aircraft  was  approaching  stall  speed 
and  overspeed  limits  simultaneously,  and  causing  wind  shear  alarms  to  sound  close 
to  landing  (Dornheim,  2005,  p.  46).  It  is  critical  for  system  reliability  that  the  software 
developers  understand  how  outputs  from  software  applications  are  used  by 
interfaced  systems  so  that  appropriate  reliability  safeguards  can  be  engineered  into 
the  developed  software. 

Software  that  freezes  or  shuts  down  the  system  when  an  anomaly  occurs  is 
certainly  not  reliable  nor  acceptable  for  critical  weapon  systems;  yet,  these 
characteristics  are  prevalent  in  commercially  based  software  systems.  Mission 
reliability  is  a  function  of  the  aggregation  of  the  system’s  subcomponent  reliability,  so 
every  software  subcomponent  is  contributing  to  or  detracting  from  that  reliability. 

The  complexity  of  software  makes  understanding  all  failure  modes  nearly 
impossible,  but  there  are  many  techniques  that  software  developers  can  employ 
when  designing  the  architecture  and  engineering  the  applications  to  improve  the 
software  component  reliability.  Once  requirements  are  clearly  communicated  to  the 
developers,  the  software  can  be  engineered  with  redundancy  or  “safe  mode” 
capabilities  to  vastly  improve  mission  reliability  when  anomalies  occur.  The  key  is 
identifying  the  reliability  requirements  and  making  them  clear  to  the  software 
developers. 
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F.  Safety  &  Security 

Very  few  software  applications  have  the  required  safety  margins  associated 
with  critical  weapon  systems  used  by  warfighters  in  combat  situations — where  they 
are  depending  on  these  margins  for  their  survival.  Typically,  the  software 
developers  have  only  a  vague  idea  of  what  their  software  is  doing  and  how  critical 
that  function  is  to  the  warfighter  employing  the  weapon  system.  Safety  performance 
must  be  communicated  to  the  software  developers  from  the  beginning  of 
development  so  they  have  the  link  between  software  functionality  and  systems 
safety.  For  example,  suppose  a  smart  munition  senses  that  it  does  not  have  control 
of  a  critical  directional  component,  and  it  calculates  that  it  cannot  hit  the  intended 
target.  The  next  set  of  instructions  the  software  provides  to  the  malfunctioning 
system  may  well  be  critical  to  the  safety  of  friendly  troops,  so  software  developers 
must  have  the  necessary  understanding  of  operational  safety  to  decide  how  to  code 
the  software  for  what  will  happen  next. 

Software  safety  is  clearly  linked  with  reliability  since  software  that  is  more 
reliable  is  inherently  safer.  It  is  critical  that  the  software  developer  understands  how 
the  warfighter  expects  the  software  to  operate  in  abnormal  situations,  in  degraded 
modes,  and  when  inputs  are  outside  of  expected  values.  Much  commercially  based 
software  simply  ceases  to  function  under  these  conditions  or  gives  error  messages 
that  supersede  whatever  function  was  being  performed,  none  of  which  are 
acceptable  in  combat  operations. 

With  software  performing  so  many  critical  functions,  there  is  little  doubt  that 
software  applications  are  a  prime  target  for  anyone  opposing  U.S.  and  Allied  forces. 
Critical  weapon  system  and  networking  software  must  be  resistant  to  hacking, 
spoofing,  mimicking,  and  all  other  manner  of  attack.  There  must  be  capabilities  for 
isolating  attacks  and  portions  of  networks  that  have  been  compromised  without 
losing  the  ability  to  continue  operations  in  critical  combat  situations.  The  software 
developer  must  know  that  all  of  these  capabilities  are  essential  before  he  or  she 
constructs  software  architectures  and  software  programs,  as  this  knowledge  will  be 
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very  influential  for  the  software  design  and  application  development.  The  Software 
Engineering  Institute’s  Quality  Attribute  Workshop  states,  “As  an  example,  consider 
security.  It  is  difficult,  maybe  even  impossible,  to  add  effective  security  to  a  system 
as  an  afterthought.  Component  as  well  as  communication  mechanisms  and  paths 
must  be  designed  or  selected  early  in  the  lifecycle  to  satisfy  security  requirements” 
(Barbacci  et  al. ,  2003,  p.  2). 

Interoperability  challenges  are  increased  when  the  SoS  has  the  type  of 
security  requirements  needed  by  the  DoD.  Legacy  systems  and  existing  security 
protocols  will  likely  need  to  be  considered  before  other  security  architecture  can  be 
effectively  designed.  OA  capabilities  will  be  hampered  by  the  critical  need  for 
security;  both  must  be  carefully  balanced  to  optimize  system  performance  and 
security.  This  balance  of  OA  and  security  must  be  managed  by  the  DoD  and  not  the 
software  developer. 

Physical  security  schemes  and  operating  procedures  will  also  have  an  impact 
on  the  software  architecture.  For  example,  many  communication  security 
(COMSEC)  devices  need  only  routine  security  until  the  keys,  usually  software 
programs,  are  applied;  then,  much  more  stringent  security  procedures  are 
implemented.  Knowledge  of  this  security  feature  would  be  a  key  requirement  of  the 
developer;  he  or  she  must  understand  how  and  when  the  critical  software  pieces  are 
uploaded  to  the  COMSEC  device.  The  same  holds  true  for  weapon  systems  that 
upload  sensitive  mission  data  just  prior  to  launch. 

Residual  software  on  equipment  or  munitions  that  could  fall  into  enemy  hands 
presents  another  type  of  security  challenge  that  needs  to  be  addressed  during  the 
application  development.  For  example,  the  ATACMS  missile  air-delivers  some  of  its 
warheads,  leaving  the  missile  body  to  freefall  to  the  surface.  It  is  very  conceivable 
that  the  body  could  be  intact  and,  of  course,  unsecured.  If  critical  mission  software 
was  still  within  the  body  and  found  by  enemy  forces,  valuable  information  might  be 
gleaned  from  knowing  how  the  system  finds  its  targets.  The  Government  would 
certainly  want  the  developer  to  design  the  applications  in  a  way  that  would  make 
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anything  recovered  useless  to  the  enemy,  but  this  is  a  capability  that  is  not  intuitive 
to  the  software  developers  (Naegle,  2006,  pp.  17-25). 

G.  Effective  Software  Development  Tools  Supporting  System 
TOC  Analyses 

1.  Software  Engineering  Institute’s  (SEI)  Quality  Attribute  Workshop 
(QAW) 

The  QAW  is  designed  to  help  identify  a  complete  (or  as  complete  as  possible) 
inventory  of  system  software  requirements  through  analysis  of  system  quality 
attributes.  One  of  the  intents  is  to  develop  the  derived  and  implied  requirements 
from  the  user-stated  requirements,  which  is  a  necessary  step  when  user-stated 
requirements  are  provided  in  terms  of  capabilities  needed  as  prescribed  by  the  Joint 
Capabilities  Integration  Development  System  (JCIDS)  process.  A  system’s  TOC, 
and  those  elements  that  contribute  to  TOC,  are  system  quality  attributes.  Although 
obviously  important  to  the  warfighter,  the  associated  operations  and  support, 
training/education,  and  facility  costs  are  rarely  addressed  in  much  detail  and  need  to 
be  derived  from  stated  requirements  or  augmented  with  implied  requirements 
through  the  QAW  process,  or  something  similar. 

The  QAW  helps  provide  a  facilitating  framework  and  process  designed  to 
more  fully  develop  the  derived  and  implied  requirements  that  are  critical  to  clearly 
communicate  to  potential  contractors  and  software  developers.  Including  a  robust 
LSA  process  using  the  MUIRS  focus  elements,  described  previously,  within  the 
QAW  process  will  likely  significantly  improve  requirements  analysis  for  those 
associated  TOC  elements  and  vastly  improve  the  accuracy  of  system  TOC 
projections.  While  improving  the  system  requirements  development,  QAW  is 
designed  to  work  with  another  SEI  process  called  the  Architectural  Tradeoff  Analysis 
MethodologySM  (ATAMsm)  to  further  improve  the  understanding  of  the  system  for 
potential  contractors  and  software  developers. 
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H.  SEI’s  Architectural  T radeoff  Analysis  MethodologySM 

(ATAMsm) 

The  Software  Engineering  Institute’s  Architectural  Trade-off  Analysis 
MethodologySM  (ATAMsm)  is  an  architectural  analysis  tool  designed  to  evaluate 
design  decisions  based  on  the  quality  attribute  requirements  of  the  system  being 
developed.  The  methodology  is  a  process  for  determining  whether  the  quality 
attributes,  including  TOC  attributes,  are  achievable  by  the  architecture  as  it  has  been 
conceived  before  enormous  resources  have  been  committed  to  that  design.  One  of 
the  main  goals  is  to  gain  insight  into  how  the  quality  attributes  trade-off  against  each 
other  (Kazman,  Klein,  &  Clements,  2000,  p.  1). 

Within  the  Systems  Engineering  Process  (SEP),  the  ATAMsm  provides  the 
critical  requirements  loop  process,  tracing  each  requirement  or  quality  attribute  to 
corresponding  functions  reflected  in  the  software  architectural  design.  Whether 
ATAMsm  or  another  analysis  technique  is  used,  this  critical  SEP  process  must  be 
performed  to  ensure  that  functional-  or  object-oriented  designs  meet  all  stated, 
derived,  and  implied  warfighter  requirements.  In  complex  systems  development 
such  as  weapon  systems,  half  or  more  than  half  of  the  total  software  development 
effort  will  be  expended  in  the  architectural  design  process.  Therefore,  the  DoD  PMs 
must  ensure  that  the  design  is  addressing  requirements  in  context  and  that  the 
resulting  architecture  has  a  high  probability  of  producing  the  warfighters’  JCIDS 
stated,  derived,  or  implied  requirements. 

The  ATAMsm  focuses  on  quality  attribute  requirements,  so  it  is  critical  to  have 
precise  characterizations  for  each.  To  characterize  a  quality  attribute,  the  following 
questions  must  be  answered: 

What  are  the  stimuli  to  which  the  architecture  must  respond? 

What  is  the  measurable  or  observable  manifestation  of  the  quality 
attribute  by  which  its  achievement  is  judged? 
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■  What  are  the  key  architectural  decisions  that  impact  achieving  the 
attribute  requirement?  (2000,  p.  5) 

The  ATAMsm  scenarios  are  a  key  to  providing  the  necessary  information  to 
answer  the  first  two  questions,  driving  the  software  engineer  to  design  the 
architecture  to  answer  the  third.  This  is  a  critical  point  at  which  all  of  the  MUIRS 
elements  need  to  be  considered  and  appropriate  scenarios  developed. 

The  ATAMsm  uses  three  types  of  scenarios:  Use-case  scenarios  involve 
typical  uses  of  the  system  to  help  understand  quality  attributes  in  the  operational 
context;  growth  scenarios  involve  anticipated  design  requirements,  including 
upgrades,  added  interfaces  supporting  SoS  development,  and  other  maturity  needs; 
and  exploratory  scenarios  involve  extreme  conditions  and  system  stressors, 
including  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA)  scenarios  (2000, 
pp.  13-15).  As  depicted  in  Figure  2,  the  scenarios  build  on  the  basis  provided  in  the 
JCIDS  documents  and  requirements  developed  through  the  QAW  process.  These 
processes  lend  themselves  to  development  in  an  Integrated  Product  Team  (IPT) 
environment  led  by  the  user/combat  developer  and  including  all  of  the  system’s 
stakeholders.  The  IPT  products  will  include  a  set  of  scenarios,  prioritized  by  the 
needs  of  the  warfighter  for  system  capability.  The  prioritization  process  provides  a 
basis  for  architecture  trade-off  analyses.  When  fully  developed  and  prioritized,  the 
scenarios  provide  a  more  complete  understanding  of  requirements  and  quality 
attributes  in  context  with  the  operation  and  support  (including  all  of  the  MUIRS 
elements)  of  the  system  over  its  life  cycle.  A  more  complete  understanding  of  the 
system’s  TOC  elements  should  emerge  from  this  type  of  analysis. 
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Figure  2.  QAW  &  ATAMsm  integration  into  Software  Life  Cycle  Management 

Just  as  the  QAW  process  provides  a  methodology  supporting  RFP,  source- 
selection  activities,  and  the  Software  Specification  and  System  Requirements 
Reviews  (SSR  and  SRR),  the  ATAMsm  provides  a  methodology  supporting  design 
analyses,  test  program  activities,  and  the  System  Functional  and  Preliminary  Design 
Reviews  (SFR  and  PDR).  The  QAW  and  ATAMsm  methodologies  are  probably  not 
the  only  effective  methods  supporting  software  development  efforts,  but  they  fit 
particularly  well  with  the  DoD’s  goals,  models,  and  SEP  emphasis.  The  user/combat 
developer  (blue  arrow  block  in  Figure  2)  is  kept  actively  involved  throughout  the 
development  process — providing  key  insights  the  software  developer  needs  to 
successfully  develop  warfighter  capabilities  in  a  sustainable  design  for  long-term 
effectiveness  and  suitability.  The  system  development  activities  are  conducted  with 
superior  understanding  and  clarity,  reducing  scrap  and  rework,  and  saving  cost  and 
schedule.  The  technical  reviews  and  audits  (part  of  the  DoD  overarching  SEP)  are 
supported  with  methodologies  that  enhance  both  the  visibility  of  the  necessary 
development  work  as  well  as  the  progress  toward  completing  it. 
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One  of  the  main  goals  in  analyzing  the  scenarios  is  to  discover  key 
architectural  decision  points  that  pose  risks  for  meeting  quality  requirements. 
Sensitivity  points  are  determined,  such  as  real-time  latency  performance  shortfalls  in 
target  tracking.  Trade-off  points  are  also  examined  so  that  TOC  impacts  resulting 
from  proposed  trade-offs  can  be  analyzed.  The  Software  Engineering  Institute 
explained,  “Trade-off  points  are  the  most  critical  decisions  that  one  can  make  in  an 
architecture,  which  is  why  we  focus  on  them  so  carefully”  (Kazman  et  al.,  2000,  p. 
23). 

The  ATAMsm  provides  an  analysis  methodology  that  complements  and 
enhances  many  of  the  key  DoD  acquisition  processes.  It  provides  the  requirements 
loop  analysis  in  the  SEP,  extends  the  user/stakeholder  JCIDS  involvement  through 
scenario  development,  provides  informed  architectural  trade-off  analyses,  and  vastly 
improves  the  software  developer’s  understanding  of  the  quality  requirements  in 
context.  Architectural  risk  is  significantly  reduced,  and  the  software  architecture 
presented  at  the  Preliminary  Design  Review  (PDR)  is  likely  to  have  a  much  higher 
probability  of  meeting  the  warfighters’  need  for  capability,  including  TOC  elements. 

Together,  the  QAW  and  ATAMsm  provide  effective  tools  for  addressing 
problem  areas  common  in  many  DoD  software-intensive  system  developments: 
missing  or  vaguely  articulated  performance  requirements,  significantly 
underestimated  software  development  efforts  (resulting  in  severely  underestimated 
schedules  and  budgets),  and  poor  communication  between  the  software  developer 
and  the  Government  (both  user  and  PM).  Both  tools  provide  frameworks  for  more 
detailed  requirements  development  and  more  effective  communication,  but  they  are 
just  tools — by  themselves,  they  will  not  replace  the  need  for  sound  planning, 
management  techniques,  and  effort.  Both  QAW  and  ATAMsm  provide 
methodologies  for  executing  SEP  Requirements  Analysis  and  Requirements  Loop 
functions,  effective  architectural  design  transition  from  user  to  developer,  and  SEP 
design  loop  and  verification  loop  functions  within  the  Test-case  Development. 
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A  significant  product  resulting  from  the  ATAMsm  is  the  development  of  test 
cases  correlating  to  the  use  case,  growth,  and  exploratory  scenarios  developed  and 
prioritized.  Figure  3  depicts  the  progression  from  user-stated  capability 
requirements  in  the  JCIDS  documents  to  the  ATAMsm  scenario  development,  and 
finally  to  the  corresponding  test  cases  developed.  The  linkage  to  the  user 
requirements  defined  in  the  JCIDS  documents  is  very  strong  as  those  documents 
drive  the  development  of  the  three  types  of  scenarios,  and,  in  turn,  the  scenarios 
drive  the  development  of  the  use  cases.  The  prioritization  of  the  scenarios  from 
user-stated  Key  Performance  Parameters  (KPPs),  Critical  Operational  Issues 
(COIs),  and  FMECA  analysis  flows  to  the  test  cases,  helping  to  create  a  system  test 
program  designed  to  focus  on  effectiveness  and  suitability  tests — culminating  in  the 
system  Operational  Test  and  Evaluation  (OT&E).  FMECA  is  one  of  the  focus  areas 
that  will  have  a  dynamic  impact  on  TOC  analysis  because  it  will  help  identify 
software  components  that  need  higher  reliability  and  back-up  capability.  The  MUIRS 
focus  helps  ensure  that  TOC  elements  are  addressed  in  design  and  test. 

The  traceability  from  user-stated  requirements  through  scenario  development 
to  test-case  development  provides  a  powerful  communication  and  assessment 
methodology.  The  growth  scenarios  and  resulting  test  cases  are  particularly  suited 
for  addressing  and  evaluating  TOC  design  requirements  because  the  system 
evolves  over  its  life  cycle,  which  is  often  overlooked  in  current  system  development 
efforts. 
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Figure  3.  Capabilities-Based  AT  AM  Scenario  Development 

The  software  developer’s  understanding  of  the  eventual  performance  required 
in  order  to  be  considered  successful  guides  the  design  of  the  architecture  and  every 
step  of  the  software  development,  coding,  and  testing  through  to  the  Full  Operational 
Capability  (FOC)  delivery  and  OT&E.  Coding  and  early  testing  of  software  units  and 
configuration  items  is  much  more  purposeful  due  to  this  level  of  understanding.  The 
MUIRS  and  FMECA  focus  will  help  the  design  process  for  better  TOC  performance. 

The  resulting  test  program  is  very  comprehensive  as  each  prioritized  scenario 
requires  testing  or  other  verification  methodologies  to  demonstrate  how  the  software 
performs  in  each  related  scenario  and  satisfies  the  quality  attributes  borne  of  the 
user  requirements.  The  testing  supports  the  SEP  design  loop  by  verifying  that  the 
software  performs  the  functions  allocated  to  it  and,  in  aggregate,  performs  the 
verification  loop  process  by  demonstrating  that  the  final  product  produces  the 
capability  identified  in  the  user  requirements  through  operational  testing. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-58- 


Both  QAW  and  ATAMsm  require  the  capturing  of  essential  data  supporting 
decision-making  and  documenting  decisions  made.  These  databases  would  be  best 
used  in  a  collaborative  IT  system,  as  described  in  the  next  section. 

I.  Collaborative  IT  Systems 

Collaborative  IT  tools  are  being  used  today  in  the  private  sector  to  connect 
various  stakeholders — designers,  logisticians,  cost  analysts,  field  service 
representatives,  system  users — who  have  the  need  to  communicate.  Such  tools 
could  be  used  to  support  current  and  emerging  warfighting  systems.  Collaborative 
tools  could  be  adapted  to  address  reliability  and  ownership  cost  concerns  related  to 
warfighting  systems.  Tools  that  facilitate  improved  communications  would  likely 
have  immediate  payoff  in  being  able  to  speed  up  solutions  to  problems.  For 
example,  field  service  representatives  (FSRs)  and  users  could  quickly  raise 
problems  to  technical  staff  for  resolution.  Cost  analysts  could  more  quickly  identify 
emerging  cost  drivers  and  initiate  business  case  analyses.  Production  and  quality 
technicians  could  rapidly  learn  of  field  defects  that  are  the  result  of  production 
defects.  Other  FSRs  and  users  could  be  alerted  to  emerging  problems  and  be 
armed  with  advance  knowledge  that  might  avert  impending  failures. 

The  reliability  improvement  process  could  be  enhanced  by  the  use  of 
collaborative  tools,  because  of  the  ease  with  which  LCL  professionals  could  bring 
repair  parts  databases  to  bear  on  design  decisions.  This  would  be  helped  by 
Pareto,  that  is,  a  focus  on  the  cost  drivers  or  reliability  drivers,  especially  the 
expensive  items  that  fail  more  often  than  predicted.  This  approach  could  be  used 
up-front  in  pre-acquisition  phases,  too,  by  tying  in  legacy  databases  that  contain 
performance  information  of  similar  or  predecessor  systems. 

Think  of  the  impact  to  business  case  analysis  (BCA).  Cost  estimates  depend 
on  solid  cost  databases  that  are  continually  updated  by  current  systems  in  order  to 
identify  major  cost  drivers  that  might  be  candidates  for  redesign  or  improved 
manufacturing  processes  to  achieve  better  reliability  and  reduced  life  cycle  cost. 
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Collaborative  IT  could  contribute  to  the  accuracy  and  completeness  of  cost 
estimates. 

Component  improvements  that  result  from  collaborative  databases  would  pay 
off  in  legacy  systems,  but  might  deliver  a  second  payoff  in  reduced  ownership  cost 
of  future  systems  as  well.  Collaborative  databases  could  be  cross-referenced  in  an 
architecture  that  would  arrange  cost  and  reliability  information  in  system,  subsystem, 
or  component  databases,  enabling  better  cost  estimating  of  emerging  systems. 

An  example  of  the  potential  value  of  collaborative  efforts  in  improving 
reliability  and  reducing  TOC  is  the  microwave  tube  on  the  Aegis  program,  developed 
in  the  early  1980s.  The  tubes  were  expensive  to  maintain  (an  estimated  $8.20  per 
operating  hour),  ubiquitous  (nearly  30,000  units  in  2010),  and  initial  reliability 
numbers  were  lower  than  expected  (as  low  as  1300  hours  MTBF).  Through  a 
collaborative  effort  between  the  Program  Manager,  NAVSEA,  and  several 
commercial  vendors,  design  and  manufacturing  improvements  increased  the  MTBF 
to  40,000  to  45,000  hours,  drastically  reducing  the  associated  TOC  from  $8.20  to 
$0.45  per  operating  hour  for  all  associated  Naval  combat  systems.  (Apte  & 
Dutkowski,  May  2006,  pp  3  -21 ) 

Collaborative  IT  tools  could  potentially  be  implemented  through  apps  to  smart 
handheld  devices,  such  as  iPhones,  Androids,  or  Blackberries.  These  devices, 
which  are  ubiquitous  at  systems  commands  and  contractor  design  and  logistics 
facilities,  could  be  very  valuable  and  convenient  for  field  service  representatives, 
military  maintenance  personnel,  and  even  users  in  some  environments. 

Very  possibly,  collaborative  IT  tools  are  in  use,  contributing  to  better  data  and 
faster  solutions  to  service  member  problems  on  legacy  systems.  On  its  face,  the 
DoD  needs  to  embrace  such  tools  to  improve  the  flow  of  technology,  acquisition,  and 
logistics  information. 
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VII. 


Conclusions  and  Recommendations:  Major 
Thrusts  to  Control  TOC 


Many  of  the  TOC  initiatives  implemented  since  our  last  TOC  research  report 
in  2003  are  definitely  steps  in  the  right  direction  for  understanding,  assessing,  and, 
ultimately,  reducing  the  TOC  financial  burden.  In  this  research,  we  have  identified 
several  areas  that  remain  as  significant  hindrances  to  effective  TOC  assessment 
and  reduction,  including  conflicting  policy  guidance,  inadequate  or  missing 
databases,  and  inadequate  process  controls  for  software  and  SoS/net-centric  TOC 
drivers.  Future  policy  and  guidance  should  address  these  shortfalls  to  more  fully 
address  TOC  issues. 

A.  Controls 

Cost  Estimates.  The  DoD  has  not  yet  demonstrated  its  ability  to  estimate 
program  costs  within  reasonable  confidence  limits.  Estimation  of  developmental 
costs  are  challenging  at  best  and  are  not  yet  well-enough  supported  by  solid  cost 
databases.  The  addition  of  O&S  cost  requirements  makes  sense  from  the 
perspective  of  life-cycle  affordability,  but  again,  this  effort  is  not  supported  by 
sufficient  O&S  cost  databases.  The  development  of  SoS  and  net-centric  systems 
exacerbates  the  cost-estimating  problem  as  system-wide  changes  drive  platform 
costs,  but  may  not  be  attributable  to  the  platform  absorbing  the  cost.  Platform 
changes  may  also  drive  system-wide  changes,  again  driving  costs  that  are  not 
attributable  to  the  system  level.  While  these  costs  may  not  be  attributable,  we 
recognize  that  they  still  need  to  be  tracked  so  that  they  can  be  estimated  in  future 
developments  and  so  that  root-cause  analyses  can  be  applied  to  help  eliminate  the 
sources  in  the  future. 

Certifications  at  MS  A  and  MS  B.  The  certifications  at  Milestones  A  and  B, 
along  with  the  attention  of  Director  CAPE,  undoubtedly  bring  attention  and  scrutiny 
to  program  cost  estimates  and  concerns  regarding  program  affordability  in  the 
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context  of  the  larger  warfighting  portfolio.  The  mandate  for  cost  certificates  is  a 
major  improvement,  as  compared  to  our  2003  research.  Cost  certificates  are  a 
necessary  forcing  function  to  push  the  DoD  toward  more  reliable  cost  estimating. 
Again,  SoS  and  net-centric  system  development  may  add  certification  challenges  as 
the  associated  costs  are  typically  not  foreseeable,  and  attributing  the  costs  to  a 
specific  PM  may  be  difficult. 

Changes  to  Nunn-McCurdy  to  include  an  O&S  Cost  metric.  Unquestionably, 
Nunn-McCurdy  requirements  have  become  more  demanding  and  onerous.  As 
challenging  as  acquisition  costs  (APUC  and  PAUC)  are,  they  are  not  the  correct 
metrics  when  viewed  from  a  life  cycle  cost  perspective.  Nunn-McCurdy  metrics 
need  to  evolve  into  measures  of  life  cycle  cost,  including  O&S  cost  portion  (e.g., 
average  O&S  cost  per  system  per  hour  or  average  O&S  cost  per  system  per  mile). 
To  do  otherwise  is  to  encourage  poor  system  development  choices  that  may  add  to 
life  cycle  cost  rather  than  constrain  it. 

Mandated  Reviews.  Moving  the  PDR  Assessment  to  precede  or  coincide 
with  Milestone  B,  as  mandated  in  WSARA,  should  improve  decision-making.  That 
is,  required  warfighter  capabilities,  technological  maturity,  affordable  resources,  and 
available  schedule  must  be  compatible  with  the  system  specification  at  Milestone  B. 
This  cannot  be  properly  assured  without  completion  of  the  preliminary  design 
because  PDR  supports  preparation  of  resource  and  schedule  estimates.  To  that 
end,  we  recommend  that  software-intensive  systems  employ  the  SEI’s  QAW  and 
ATAMsm  process  tools  (or  similar-type  processes)  to  accomplish  the  following:  more 
fully  define  derived  and  implied  software-related  requirements;  improve  the  software 
developer’s  understanding  of  how  the  warfighters  use  and  maintain  the  system; 
understand  how  the  system  is  likely  to  be  changed,  modified,  or  made  interoperable 
over  its  life  cycle;  and  improve  the  developer’s  understanding  of  the  performance  the 
warfighter  expects  under  stressful  or  unusual  operating  scenarios.  These  process 
tools  should  vastly  improve  the  reliability  of  information  resulting  from  the  PDR  with 
regard  to  the  software  components. 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  62  - 

NAVAL  POSTGRADUATE  SCHOOL 


Technological  Maturity.  The  Technology  Readiness  Assessment  Deskbook 
was  published  in  2003  and  has  greatly  clarified  understanding  of  technological 
maturity,  yet  it  is  difficult  to  apply  to  software  development.  The  DoD  has  a  long 
track  record  of  moving  into  detailed  design  after  Milestone  B  without  the  necessary 
maturity  of  technology  to  complete  the  system  design.  The  result  is  almost  always 
program  delays  and  substantial  cost  growth.  Lack  of  technological  maturity  is  one  of 
the  major  causes  of  cost  growth  and  reflects  the  importance  of  Knowledge  Point  1 
as  described  by  the  GAO.  Because  software  development  defies  early  maturity 
estimation,  it  must  be  considered  separately  and  include  the  maturity  evaluations  of 
the  software  developer  (CMMI  or  equivalent),  as  well  as  the  maturity  evaluations  of 
the  materiel  developer/PM  (SA-CMM  or  equivalent). 

Today,  we  have  a  useable  template  to  discuss  and  reach  a  common 
understanding  of  technological  maturity;  we  know  the  importance  of  technological 
maturity;  we  have  a  mandated  certification — in  law  and  regulation — to  assure  the 
intersection  of  technological  maturity,  affordability,  available  budget,  and  schedule. 
The  DoD  knows  the  elements  of  knowledge  that  are  necessary  for  sound  decision¬ 
making  to  launch  development  of  a  new  warfighting  system.  This  also  applies  to 
COTS  or  GOTS  software,  but  software  development  depends  on  assessing  the 
maturity  of  the  developer  and  the  PM  office,  as  stated  previously. 

Navy  Gate  Reviews.  The  DoD  should  require  gate  reviews  for  use  by  all  the 
Services.  Gate  reviews  provide  for  oversight  and  governance  of  MDAP  life  cycle 
cost.  These  reviews  establish  a  process  to  bring  attention  to  ownership  cost 
throughout  the  developmental  cycle  of  warfighting  systems.  In  a  wider  sense,  gate 
reviews  provide  a  forum  for  lessons  learned  regarding  TOC.  While  emphasizing 
affordability  through  the  developmental  and  production  phases  of  individual 
warfighting  systems,  gate  reviews  provide  the  opportunity  to  balance  the  resources 
provided  among  capability  portfolios,  and  potentially  to  assist  in  balancing  resources 
across  all  of  the  department’s  family  of  capability  portfolios. 
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Configuration  Steering  Boards.  The  opportunity  to  grow  requirements  for 
ongoing  programs  that  are  beyond  Milestone  B  has  been  largely  taken  away  from 
the  user  community  and  placed  into  the  hands  of  each  Service’s  Configuration 
Steering  Board.  This  is  likely  to  curtail  major  cost  increases  in  programs  and 
encourages  cost  reductions  based  on  PM  recommendations  in  program 
requirements  and  within  program  objectives.  Congressional  language  on  changes 
to  user  requirements  has  been  accommodated  in  the  most  recent  version  of  DoDI 
5000.02,  dated  December  2,  2008.  Implementation  of  this  guidance  entails  a  major 
change  in  culture;  whether  it  is  successful  in  reducing  ownership  cost  will  be  shown 
over  time. 

B.  Databases 

Defense  Acquisition  Management  Information  Retrieval  (DAMIR) — MDAP 
Systems  Database.  The  DAMIR  database  is  a  “virtual”  repository  used  by  the 
acquisition  community  and  others  to  manage  MDAP  and  MAIS  systems  and  to 
provide  relevant  information  about  those  systems  across  the  DoD.  The  database 
arrays  Selected  Acquisition  Reports  (SAR),  Defense  Acquisition  Executive  Summary 
(DAES)  reports,  Acquisition  Program  Baselines  (APBs),  and  SAR  Baselines.  It 
contains  other  program  information,  such  as  missions  and  descriptions,  system 
performance,  schedules,  cost  and  funding  (including  operations  and  support  costs), 
Nunn-McCurdy  breaches,  contracts  performance,  and  manufacturing  and  deliveries. 
DAMIR  contains  some  capability  to  compare  programs  in  terms  of  cost  and  schedule 
performance  and  to  summarize  cost  and  schedule  information  (e.g.,  by  warfighting 
system  or  Service). 

VAMOSC  databases  that  collect  O&S  cost  information  should  be  improved  or 
replaced  for  better  support  of  cost  estimating.  Current  GAO  reports  indicate  that 
VAMOSC  is  inaccurate,  incomplete,  and  internally  inconsistent.  VAMOSC  should  be 
able  to  provide  data  on  similar  or  predecessor  systems,  subsystems,  and 
components  in  support  of  programs  in  development,  in  addition  to  providing  accurate 
O&S  cost  performance  for  legacy  systems  in  their  sustainment  phase. 
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Software  component  analysis  and  decision  databases,  like  those  that  would 
be  developed  using  the  QAW  and  ATAMsm  tools,  should  be  required  for  every 
software-intensive  system.  Software  continues  to  be  a  “wildcard”  in  estimating  both 
acquisition  costs  and  O&S  costs,  so  front-end  analyses  must  be  improved, 
cataloged,  and  shared  widely  through  a  collaborative  environment. 

Collaborative  databases  to  gather  enterprise/system/subsystem/component 
cost  information  should  be  established  to  facilitate  collaboration  among  experts  who 
are  widely  dispersed.  One  can  envision  collaborative  IT  systems  being  employed  by 
systems  commands  and  the  DLA.  Such  systems  could  support  national-level 
enterprise  requirements  at  one  end  of  the  spectrum  or  components  at  the  opposite 
end.  In  any  case,  collaborative  IT  systems  could  be  set  up  for  broad  sharing  of 
information  that  might  be  useful  to  developers  of  new  systems,  to  maintainers  of 
legacy  systems,  or  to  O&S  cost  analysts  trying  to  improve  the  performance  of 
components  that  are  cost  drivers. 

C.  Performance  Based  Logistics 

The  DoD  is  very  familiar  with  the  demands  of  sustainment — but  the  OSD  has 
not  insisted  on  proper  planning  and  implementation  of  affordable  sustainment.  The 
OSD  has  not  focused  enough  on  the  metrics  that  indicate  success  of  warfighting 
systems  or  on  the  cost  to  achieve  required  metrics.  Instead,  focus  has  been  on 
commodity  management,  with  the  DLA  being  a  prime  example,  where  metrics  have 
reflected  performance  of  the  support  organization,  but  not  weapon  system 
readiness. 

PBL  must  be  applied  more  widely,  such  that  non-PBL  systems  should  be  an 
unusual  occurrence.  PBL  requirements  initially  should  be  analyzed  vertically  by  an 
individual  system  such  that  the  warfighting  system  is  able  to  achieve  its  mission  and 
is  affordable.  However,  PBL  arrangements  also  should  be  analyzed  horizontally  to 
take  advantage  of  economic  quantities  and  other  efficiencies  that  might  be  provided 
by  using  common  support  systems.  PBL  metrics  also  should  be  devised  to  reflect 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  65  - 

NAVAL  POSTGRADUATE  SCHOOL 


the  individual  warfighting  system  (i.e. ,  vertical)  and  the  broader  support  system  or 
enterprise  (i.e.,  horizontal). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-66- 


References 


Apte,  A.  &  Dutkowski,  E.  (2006,  May  31J.  Total  ownership  cost  reduction  case 
study:  AEGIS  microwave  power  tubes  (NPS-AM-06-008).  Retrieved  from 
Naval  Postgraduate  School,  Acquisition  Research  Program  website: 
http://www.acquisitionresearch.net 

Barbacci,  M.,  Ellison,  R.,  Lattanze,  A.,  Stafford,  J.,  Weinstock,  C.,  &  Wood,  W. 

(2003,  August).  Quality  attribute  workshops  (QAWs)  (3rd  ed.;  CMU/SEI-2003- 
TR-016).  Pittsburgh,  PA:  Carnegie  Mellon  University,  Software  Engineering 
Institute. 

Boudreau,  M.  W.,  &  Naegle,  B.  R.  (2003,  September  30).  Reduction  of  total 
ownership  cost  (NPS-AM-03-004).  Retrieved  from  Naval  Postgraduate 
School,  Acquisition  Research  Program  website: 
http://www.acquisitionresearch.net 

Brodsky,  R.  (2010,  May  19).  Pentagon  reports  progress  on  anniversary  of 
procurement  reform  law.  Retrieved  from  http://www. GovExec. com 

Business  Executives  for  National  Security  (BENS),  Task  Force  on  Defense 

Acquisition  Law  and  Oversight.  (2009,  July).  Getting  to  best:  Reforming  the 
defense  acquisition  enterprise.  Washington,  DC:  BENS. 

Carnegie  Mellon  University,  Software  Engineering  Institute.  (2007).  The  importance 
of  software  architecture.  Retrieved  from 
http://www.sei.cmu.edu/architecture/index.html 

Chairman  of  the  Defense  Science  Board  (DSB).  (2008,  May).  Report  of  the  Defense 
Science  Board  Task  Force  on  developmental  test  &  evaluation.  Washington, 
DC:  Author. 

Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS).  (2009a,  March  1 ).  Joint  capabilities 
integration  and  development  system  (CJCS  Instruction  31 70.01  G). 
Washington,  DC:  Author. 

Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS).  (2009b,  July  31 ).  Manual  for  the 
operation  of  the  joint  capabilities  integration  and  development  system 
(CJCSM  3170.01).  Washington,  DC:  Author. 

Defense  acquisition:  Improved  program  outcomes  are  possible:  Written  testimony  of 
GAO’s  Louis  J.  Rodrigues  before  the  Subcommittee  on  Acquisition  and 
Technology  of  the  Senate  Committee  on  Armed  Services  (GAO/T-NSIAD-98- 
123),  105th  Cong.  (1998).  Retrieved  from  GAO  website:  http://www.qao.gov 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-67  - 


Defense  acquisitions:  Fundamental  changes  are  needed  to  improve  weapon 

program  outcomes:  Written  testimony  of  GAO’s  Michael  J.  Sullivan  before  the 
Subcommittee  on  Federal  Financial  Management,  Government  Information, 
Federal  Services,  and  International  Security  of  the  Senate  Committee  on 
Homeland  Security  and  Governmental  Affairs  (GAO-08-1 159T).  1 10th  Cong. 
(2008).  Retrieved  from  GAO  website:  http://www.qao.gov 

Defense  Acquisition  University  (DAU).  (2005,  March).  Performance  based  logistics: 

A  program  manager’s  product  support  guide.  Fort  Belvoir,  VA:  Author. 

Department  of  the  Navy  (DoN).  (2010,  February  16).  Total  ownership  cost 

guidebook  [Concurrent  with  SECNAVIST  5000. 2E],  Washington,  DC:  Author. 

Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  (DUSD[S&T]). 
(2005,  May).  Technology  readiness  assessment  (TRA)  deskbook. 
Washington,  DC:  Author. 

Director  of  Operational  Test  and  Evaluation  (DOT&E)  [J.  Michael  Gilmore],  (2010, 
June  30).  State  of  reliability  [Memorandum],  Washington,  DC:  Author. 

Dornheim,  M.  A.  (2005,  September).  A  wild  ride.  Aviation  Week  &  Space 
Technology,  163,  46. 

Duncan  Hunter  National  Defense  Authorization  Act  for  Fiscal  Year  2009,  Pub.  L.  No. 
110-417,  §  814,  122  Stat.  4356  (2008,  October  14). 

Eaton,  D.  R.  (2004,  August  1).  Improving  the  management  of  reliability  (NPS-LM-04- 
009).  Retrieved  from  Naval  Postgraduate  School,  Acquisition  Research 
Program  website:  http://www.acquisitionresearch.net 

F/A  18.  (201 1,  March).  In  Wikipedia.  Retrieved  from  http://www.wikipedia.org 
/wiki/McDonnell  Douglas  F/A-18  Hornet 

Fast,  W.  R.  (2011,  January-February).  WSARA  one  year  later.  Defense  AT&L,  40, 
4-8. 

Federation  of  American  Scientists  (FAS).  (2011a,  March  10).  Ml  Abrams  main  battle 
tank.  Retrieved  from  http://www.fas.org/man/dod-101/svs/land/m1  .htm 

Federation  of  American  Scientists  (FAS).  (2011b,  March  11).  F/A  18  Hornet. 
Retrieved  from 

http://www.fas.org/proqrams/ssp/man/uswpns/air/fiqhter/f18.html 
Fowler,  R.  (2010,  March-April).  The  future  of  product  support.  Defense  AT&L,  16- 


acquisition  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  68  - 

NAVAL  POSTGRADUATE  SCHOOL 


General  Accounting  Office  (GAO).  (2003,  February  11).  Best  practices:  Setting 

requirements  differently  could  reduce  weapon  systems’  total  ownership  costs: 
Report  to  the  Subcommittee  on  Readiness  and  Management  Support, 
Committee  on  Armed  Services  (GAO-03-57).  Retrieved  from 
http://www.qao.gov 

Government  Accountability  Office  (GAO).  (2008,  December).  Defense  logistics: 

Improved  analysis  and  cost  data  needed  to  evaluate  the  cost-effectiveness  of 
performance  based  logistics  (GAO-09-41).  Retrieved  from  http://www.qao.gov 

Government  Accountability  Office  (GAO).  (2010a,  May).  Defense  acquisitions: 

Strong  leadership  is  key  to  planning  and  executing  stable  weapon  programs 
(GAO  10-522).  Retrieved  from  http://www.qao.gov 

Government  Accountability  Office  (GAO).  (2010b,  July  20).  Defense  management: 
DoD  needs  better  information  and  guidance  to  more  effectively  manage  and 
reduce  operating  and  support  costs  of  major  weapon  systems  (GAO-10-717). 
Retrieved  from  http://www.qao.gov 

Hill,  S.  (2006,  September).  A  winning  strategy:  Collaborating  on  product  designs 
speeds  time-to-market,  improves  quality,  boosts  revenues.  Manufacturing 
Business  Technology,  24,  18-21. 

Humphrey,  W.  S.  (1990).  Managing  the  software  process.  New  York,  NY:  Addison- 
Wesley. 

Institute  for  Defense  Analyses  (IDA),  (n.d.).  Reduction  of  total  ownership  costs. 
Retrieved  from  http://rtoc.ida.org/rtoc/rtoc.html 

Kaye,  M.  A.,  Sobota,  M.  S.,  Graham,  D.  R.,  &  Gotwald,  A.  L.  (2000,  Fall).  Cost  as  an 
independent  variable.  Defense  Acquisition  Review  Quarterly,  24,  353-371. 

Kazman,  R.,  Klein,  M.,  &  Clements,  P.  (2000,  August).  ATAMsm:  Method  for 

architecture  evaluation  (CMU/SEI-2000-TR-004).  Pittsburgh,  PA:  Carnegie 
Mellon  University,  Software  Engineering  Institute. 

Kobren,  B.  (2010).  The  product  support  manager:  Achieving  success  in  executing 
life  cycle  management  responsibilities.  Defense  Acquisition  Review  Journal, 
54,  182-204. 

Kratz,  L.,  &  Buckingham,  B.  A.  (2010).  Achieving  outcomes-based  life  cycle 
management.  Defense  Acquisition  Review  Journal,  53,  45-66. 

Kreisher,  O.  (2010,  May  6).  Navy  secretary  outlines  how  he'll  reduce  weapons  costs. 
Retrieved  from  http://www.GovExec.com 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-69  - 


Major  defense  acquisition  programs:  Certification  required  before  milestone  A  or  key 
decision  point  A  approval,  10  U.S.C.  §  2366a  (2009). 

Major  defense  acquisition  programs:  Certification  required  before  milestone  B  or  key 
decision  point  B  approval,  10  U.S.C.  §  2366b  (2009). 

Mishory,  J.  (2010,  August  19).  CAPE  report  finds:  DoD  cannot  create  baselines  to 
track  operating  and  support  costs.  Inside  the  Pentagon.  Retrieved  from 
http://defensenewsstand.com/lnside-the-Pentaqon/ 

Morris,  E.,  Levine,  L.,  Meyers,  C.,  Place,  P.,  &  Plakosh,  D.  (2004,  April).  System  of 
systems  interoperability  (SOSI):  Final  report.  Pittsburg,  PA:  Carnegie  Mellon 
University,  Software  Engineering  Institute. 

Naegle,  B.  R.  (2004,  July).  The  impact  of  software  support  on  total  ownership  cost 
(NPS-AM-04-007).  Monterey,  CA:  Naval  Postgraduate  School. 

Naegle,  B.  R.  (2006,  September).  Developing  software  requirements  supporting 

open  architecture  performance  goals  in  critical  DoD  system-of-systems  (NPS- 
AM-06-035).  Monterey,  CA:  Naval  Postgraduate  School. 

Naegle,  B.  R.,  &  Petross,  D.  (2007,  October).  Developing  software  requirements 
supporting  open  architecture  performance  goals  in  critical  DoD  system-of- 
systems  (NPS-AM-07-104).  Monterey,  CA:  Naval  Postgraduate  School. 

Naegle,  B.  R.,  &  Petross,  D.  (2010,  January).  P-8A  Poseidon  multi-mission  maritime 
aircraft  (MMA)  software  maintenance  organization  concept  analysis  (NPS- 
LM-10-006).  Monterey,  CA:  Naval  Postgraduate  School. 

National  Defense  Authorization  Act  for  Fiscal  Year  2010,  Pub.  L.  No.  111-84,  §  805, 
123  Stat.  2190  (2009,  October  28). 

Navy  Logistics  Functional  IPT.  (2010,  April  30).  DON  acquisition  governance  (Gate 
Reviews):  Establishing  a  post-IOC  sustainment  review  [Briefing],  Washington, 
DC:  DON. 

Office  of  the  Secretary  of  Defense  (OSD).  (2003,  March  21).  [Aldridge,  E.  C.,  & 
Stenbit,  J.  P.].  Software  acquisition  process  improvement  programs. 
[Memorandum],  Washington,  DC:  Author. 

Peters,  K.  M.  (2010,  February  2).  Defense  budget  tackles  major  management 
issues.  Retrieved  from  http://www. GovExec.com 

Porter,  G.,  Gladstone,  B.,  Gordon,  C.  V.,  Karvonides,  N.,  Kneece,  R.  R.,  Jr., 

Mandelbaum,  J.,  &  O’Neil,  W.  D.  (2009,  December).  The  major  causes  of  cost 
growth  in  defense  acquisition:  Executive  Summary  (Vol.  1 ).  Alexandria,  VA: 
Institute  for  Defense  Analyses. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  70  - 

NAVAL  POSTGRADUATE  SCHOOL 


Principal  Deputy  Under  Secretary  of  Defense  (AT&L)  [Frank  Kendall],  (2010,  April 
23).  Preparation  for  Defense  Acquisition  Board  (DAB)  meetings,  DAB 
readiness  meetings  (DRM),  and  DAB  planning  meetings  (DPM) 
[Memorandum],  Washington,  DC:  USD(AT&L). 

Product  Support  Assessment  Team  (PSAT).  (2009,  November).  DoD  weapon 
system  acquisition  reform  product  support  assessment.  Washington,  DC: 
Author. 

Roper,  M.  A.  (2010).  Pre-milestone  A  cost  analysis:  Progress,  challenges,  and 
change.  Defense  Acquisition  Review  Journal,  53,  67-75. 

Scully,  M.  (2010,  February  2).  Gates  sacks  F-35  manager,  withholds  Lockheed 
payments.  Retrieved  from  http://www.GovExec.com 

Siemens.  (2010).  Knowledge-centric  MRO  transformation  (PLM  Software  White 
Paper).  Retrieved  from  http://www.siemens.com/plm 

Tremaine,  R.  L.,  &  Seligman,  D.  (2010).  It’s  time  to  take  the  chill  out  of  cost 

containment  and  re-energize  a  key  acquisition  practice.  Defense  Acquisition 
Review  Journal,  54,  242-267. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Jacques  S.  Gansler],  (1998,  November  13).  Definition  of  total  ownership  cost 
(TOC),  life  cycle  cost  (LCC),  and  the  responsibilities  of  program  managers 
[Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Jacques  S.  Gansler],  (1999,  October  26).  Software  evaluations  for  ACAT I 
programs  [Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Edward  C.  Aldridge],  (2002,  February  13).  Performance  based  logistics 
[Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Edward  C.  Aldridge],  (2003,  May  12).  The  defense  acquisition  system  (DoD 
Directive  5000.1).  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[John  J.  Young],  (2007a,  September  19).  Prototyping  and  competition 
[Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]). 
(2007b,  November  20).  The  defense  acquisition  system  (DoD  Directive 
5000.01).  Washington,  DC:  Author. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-71  - 


Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[John  J.  Young],  (2008a,  July  21).  Implementing  reliability,  availability  and 
maintainability  policy  [Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[John  J.  Young],  (2008b,  July  31).  Implementing  a  life  cycle  management 
framework  [Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]). 
(2008c,  December  8).  Operation  of  the  defense  acquisition  system  (DoD 
Instruction  5000.02).  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (2009,  November).  DoD  weapon  system  acquisition 
reform  product  support  assessment.  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (2009,  December  4).  Implementation  of  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  (Directive  Type  Memorandum  09- 
027).  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (2010a,  September  14).  Better  buying  power:  Guidance 
for  obtaining  greater  efficiency  and  productivity  in  defense  spending 
[Memorandum],  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (2010b,  October  7).  Requirements  for  life  cycle 
management  and  product  support  (Directive-Type  Memorandum  [DTM]  IQ- 
015).  Washington,  DC:  Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (2010c,  October  21).  Implementation  of  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  (Change  1  to  December  4,  2009, 
verision;  Directive-Type  Memorandum  [DTM]  09-027).  Washington,  DC: 
Author. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 
[Ashton  B.  Carter],  (201  Od,  November  3).  Implementation  directive  for  better 
buying  power — Obtaining  greater  efficiency  and  productivity  in  defense 
spending  [Memorandum],  Washington,  DC:  Author. 

Weapon  System  Acquisition  Reform  Act  of  2009,  Pub.  L.  No.  111-23,  123  Stat.  1704 
(2009). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-72- 


2003  -2011  Sponsored  Research  Topics 


Acquisition  Management 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  BCA:  Contractor  vs.  Organic  Growth 

■  Defense  Industry  Consolidation 

■  EU-US  Defense  Industrial  Relationships 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to 
Shipyard  Planning  Processes 

■  Managing  the  Services  Supply  Chain 

■  MOSA  Contracting  Implications 

■  Portfolio  Optimization  via  KVA  +  RO 

■  Private  Military  Sector 

■  Software  Requirements  for  OA 

■  Spiral  Development 

■  Strategy  for  Defense  Acquisition  Research 

■  The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 

Contract  Management 

■  Commodity  Sourcing  Strategies 

■  Contracting  Government  Procurement  Functions 

■  Contractors  in  21st-century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 

■  Past  Performance  in  Source  Selection 

■  Strategic  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  USAF  Energy  Savings  Performance  Contracts 

■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Financial  Management 


■  Acquisitions  via  Leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  the  DoD 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition 
Budgeting  Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 

Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-term  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 

Logistics  Management 

Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 

ACQUISITION  RESEARCH  PROGRAM 
GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

-  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

-  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 


A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acquisitionresearch.org 


mtswTi*  ftmacuTiAu 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CALIFORNIA  93943 


www.acquisitionresearch.org 


